We had a severe incident in our organization due to an unexpected behavior during a Nodegroup replacement.
We needed to add more storage to our node in EKS, so we expanded the volumes of the instances of our pre-configured nodegroup.
After running Pulumi Preview, we saw the a replacement for the nodegroup is needed.
I checked in many places including Pulumi Docs, how does replacement works in Pulumi.
According to this page - https://www.pulumi.com/docs/concepts/options/deletebeforereplace/“By default, to minimize downtime, Pulumi creates new instances of resources before deleting old ones”
We understood that the replacement will result in creating the new Nodegroup, and when completed, delete the old one.
Unfortunately, this didn’t happen in this order, and the Pulumi run first deleted the node group, and then eventually, created a new one.
What do you think was wrong here ?
09/04/2023, 7:33 PM
We would need more information about the code you had before, and after, and what the diff looked like to really know for sure.