https://pulumi.com logo
Join Slack
Powered by
hello! I'm using Pulumi v3.30.0 in our CI to run T...
# typescript
f
hello! I'm using Pulumi v3.30.0 in our CI to run TS Pulumi projects. Our CI uses containers to execute those project and these containers have 2 vCPUs and 4G of ram allocated (and that's the highest configuration available). I'm seeing intermittent failures (only works once every 4 times) running the preview part of the 
pulumi up
command and my hunch is that this problem is related to the memory available for the TS compilation. These errors don't have much information:
Copy code
@ previewing update....
Resources:
    1 unchanged
error: an unhandled error occurred: Program exited with non-zero exit code: -1
Has anyone experienced this before and found a solution? (which isn't bumping the memory - since in this case I cannot do)
enabling 
debug
and using 
-v9
I can see the following messages:
Copy code
deployment_executor.go:259] deploymentExecutor.Execute(...): incoming event (nil? true, Error: an unhandled error occurred: Program exited with non-zero exit code: -1)

step_executor.go:364] StepExecutor worker(-1): StepExecutor.waitForCompletion(): waiting for worker threads to exit

step_executor.go:364] StepExecutor worker(-2): worker exiting due to cancellation

step_executor.go:364] StepExecutor worker(-1): StepExecutor.waitForCompletion(): worker threads all exited

deployment_executor.go:303] deploymentExecutor.Execute(...): step executor has completed
error: an unhandled error occurred: Program exited with non-zero exit code: -1

deployment_executor.go:161] deploymentExecutor.Execute(...): exiting provider canceller

ignore.go:44] Explicitly ignoring and discarding error: rpc error: code = Canceled desc = grpc: the client connection is closing
g
I don't see anything in that debug output that screams memory issue 🤔 Also it looks like the errors are occurring in golang, not TS 😕 In any case, by default, NodeJS's runtime engine has a cap of 1300MB heap space on 64 bit systems, regardless of how much RAM is actually available. You can override it by setting the NODE_OPTIONS environment variable inline with your pulumi command (or set the envvar outside of this step by other means):
Copy code
NODE_OPTIONS="--max-old-space-size=3000" pulumi preview
Pulumi might already be overriding this value internally, or it might not be a memory issue at all -- so this additional option might not have any effect. You can read more about 
max-old-space-size
here https://stackoverflow.com/a/64409997
f
I tried setting 
NODE_OPTIONS
but still experienced failures I also noticed when running in docker desktop memory consumption goes up over 3GB during the preview and when resources are getting created drops to around 100MB, that's why I suspect the TS build process can be the problem (considering this needs to happen before resources are created/updated).
5 Views
Open in Slack
PreviousNext
Powered by