AWS had a CloudFront API issue today, which happened to strike while I was deploying. My deploy created a CloudFront distribution, but their API reported an error ("error: error creating CloudFront Distribution: DistributionAlreadyExists: The caller reference that you are using to create a distribution is associated with another distribution.").
Running the deployment again would fail, because Pulumi didn't know that the CloudFront distribution existed, and would fail when it tried to create it again.
The fix was pretty painful. I had to export the state of my test stack (which had worked the night before), extract the CloudFront distribution state there, update all that info to point to my production stack, and then copy that into the state of the production stack.
Once I did that, though, pulumi didn't try to re-create the distribution again and my deployment succeeded.
But during the outage, my deployment had removed a bunch of services and DNS records so I had an hour of downtime.
Was quite painful!!! But fortunately solvable.
WOuld have been easier to solve if I didn't have to hand-edit the cloudfront distribution object. If I didn't have an example to go from, it would have been impossible.
But I'm stll glad I'm not deploying by hand, I love pulumi don't worry ❤️