I’m hoping for some guidance regarding handling of process termination in an automation API application. Conceptually, it is somewhat similar to this example: https://github.com/pulumi/automation-api-examples/tree/main/go/pulumi_over_http
We host our service in kubernetes, and occasionally experience stacks being locked when pods are terminated. We currently set a timeout on calls to stack up, which is a bit shorter than terminationGracePeriodSeconds. We also stop creating new resources when we receive SIGTERM. This should, I believe, ensure that we’re able to stop running operations before we receive SIGKILL. I believe this still leaves us in a situation where a lock exists on the stack (at least anecdotally, that’s what I’m seeing). Would it be best practice to call cancel on the stack before we terminate? Are there other steps we should take, to ensure that we don’t end up in a situation with locked stacks, or out of sync stacks?
09/29/2022, 6:28 PM
Are you using an OSS backend? If so, the locked stacks issue will probably become an issue due to the way the consistency locking works on an object store. cancellation issue is a much trickier problem that is probably best covered in a call. Would you be opening to having a chat with a Pulumi solutions engineer to see if we can work through the use cases?
10/14/2022, 7:35 AM
Sorry for the late reply, @billowy-army-68599. Yes, we are using an OSS backend. Are there significant advantages to using Pulumi Service in this regard, and could you elaborate on what those might be?