I’m hoping for some guidance regarding handling of process termination in an automation API application. Conceptually, it is somewhat similar to this example:
https://github.com/pulumi/automation-api-examples/tree/main/go/pulumi_over_http
We host our service in kubernetes, and occasionally experience stacks being locked when pods are terminated. We currently set a timeout on calls to stack up, which is a bit shorter than terminationGracePeriodSeconds. We also stop creating new resources when we receive SIGTERM. This should, I believe, ensure that we’re able to stop running operations before we receive SIGKILL. I believe this still leaves us in a situation where a lock exists on the stack (at least anecdotally, that’s what I’m seeing). Would it be best practice to call cancel on the stack before we terminate? Are there other steps we should take, to ensure that we don’t end up in a situation with locked stacks, or out of sync stacks?