Has anyone had any thoughts or solutions on how to deal with failures that result in pending operations in the state file that block further deployments? When doing this interactively it's not the end of the world as you can go in and fix, but using the automation API to automate things, it means your stuck with failure until someone can manually go in and fix it. Is there any way to have it ignore or remove pending operations, given we do a refresh each time anyway

You can just export/import state I think?

We've got https://github.com/pulumi/pulumi/issues/4265 open to work on this, and there's some more information in the issue itself. Until that issue is closed, the best workaround is as @prehistoric-kite-30979 noted to export and import state in the workflow. Here's a couple Go code snippets shared by one of our engineers as examples: https://pkg.go.dev/github.com/pulumi/pulumi/sdk/v3/go/auto#example-Stack.Import and https://gist.github.com/EvanBoyle/9e4f9ce95a40c288274566d2c3134de0

thanks, will look at doing the export/import for now