Hi Everyone, I'm trying to find some guidance on w...
# general
Hi Everyone, I'm trying to find some guidance on what to do when a Pulumi App can't complete because its state is out of sync with what does/doesn't exist. (details in thread)
I have a Pulumi app that deploys some Azure resources that is very strict about validating config. If a config value is invalid (missing, wrong type, etc) it throws an exception, but doesn't evaluate that config until it's needed. When running earlier today, I forgot to add a new config value to a stack and as a result the Pulumi app had an exception thrown mid-run. After fixing the config and re-running I was given an error that the app could not delete a file in a blob storage because that file no longer exists. It appears that during the inital run the app did delete the file so it can replace it with a new one, but it didn't track this because of my custom exception, and so the state is wrong. Now I can't run the Pulumi App to completion and get an error saying Pulumi can't delete a file because it doesn't exist. I tried modifying the stack state using the
pulumi state delete <urn>
command. This worked for one file, but not another because there were other resources reliant on it and I didn't want to run that delete command for resource that haven't been deleted. Since this was for a dev environment, I ran
pulumi destroy
on the stack. But that's not something I'd be able to do if this ever happens to a production stack.
I will be changing my Pulumi App to check config at the start so this scenario doesn't happen again. But if this happens for some other reason, I'd like to know how to fix it.
If it matters, the exact error was:
Copy code
azure-native:storage:Blob (WebClient_appsettings.json):
    error: deleting blob "appsettings.json" (container "$web" / account "webclient237e8fad"): blobs.Client#Delete: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="BlobNotFound" Message="The specified blob does not exist.\nRequestId:41945ff9-401e-0097-0e6b-9c84a2000000\nTime:2022-07-20T19:00:52.4791020Z"
Can't you just run
pulumi refresh
to get the state fixed?
I ran that a few times and it didn't fix the issue
That's odd, so refresh thought the blob still existed but delete 404'd
Right. I had hoped that would fix the issue, but after running refresh a few times I still got the same error that the blob can't be deleted