faint-motherboard-95438
02/12/2021, 10:58 AMpulumi-aws
just trashed all my main zone without any reason (in production, else that’s not fun), while i didn’t change anything for ages related to this zone or its records (i was updating something else in the stack):
+-aws:route53/zone:Zone: (replace)
[id=[REDACTED]]
[urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
[provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
~ name: "[REDACTED]." => "[REDACTED]"
++aws:route53/zone:Zone: (create-replacement)
[id=[REDACTED]]
[urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
[provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
~ name: "[REDACTED]." => "[REDACTED]"
+-aws:route53/record:Record: (replace)
[id=[REDACTED]_TXT]
[urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/record:Record::[REDACTED]-record-3-TXT]
[provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
~ zoneId : "[REDACTED]" => output<string>
Since most of the time it just wants to update some props related to zone or records (that’s weird btw), I didn’t pay attention (my fault), but this time it trashed all my records, trying to replace them without any reason nor changes, and failed when it tried to delete the main zone (obviously since it’s not empty). Now it’s stuck deleting the records & the zone and preventing me to recreate them !
I had to quickly recreate everything by hand and now I have a stack completely stale, desync and unusable…
I suppose that’s related to the update of pulumi-aws
, but even trying to downgrade to 2.x
it still wants to delete the zone by renaming it zone.tld.
=> zone.tld
(which does not make sense to me since I didn’t put a trailing dot in the domain name in the first place nor change anything in that regard)
That’s impossible to maintain a production stack when it wants to delete / replace something as critical as your main NS zone, I don’t see the point here and it does not seem there’s any way to prevent it.
What should I have done and how can I save the stack now (without deleting the zone) ?billowy-army-68599
02/12/2021, 4:49 PM2.x
to 3.x
?
can you share the code you currently have? I think just adding a trailing dot would be enough to fix this, but not entirely surefaint-motherboard-95438
02/12/2021, 4:54 PM3.x
but we nearly never update the stack handling our main NS zone so that’s why I encountered this only today.
I can try to share the code but it’s pretty basic, one aws.route53.Zone
followed by a few aws.route53.Records
, nothing special.
As you can see the main problem is that the version change force a delete+replace on all aws.route53
resources. That’s basically impossible to even think about it in a real life production context, that would mean being stuck in 2.x
for life for a given production environment.billowy-army-68599
02/12/2021, 4:58 PM2.x
- I believe you just need to update your Zone
resource to add the trailing dot or alias the resource.faint-motherboard-95438
02/12/2021, 5:06 PMprovider
and name
(giving that now name
is the same between old and new value, with a trailing dot, don’t know why it still wants to change it) :
+-aws:route53/zone:Zone: (replace)
[id=[REDACTED]]
[urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
[provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
~ name: "[REDACTED]." => "[REDACTED]."
2.x
, I updated package.json
, cleaned up node_modules
& package-lock.json
, removed all plugins with pulumi plugins rm resource aws
then did a clean npm i
and ensured only pulumi-aws
2.13
is installed, but as soon as I pulumi up
, it installs the 3.23
plugin before the preview, don’t know whyimportant-appointment-55126
02/12/2021, 5:51 PMdelegationSetId
billowy-army-68599
02/12/2021, 5:52 PMignoreChanges
for that field?important-appointment-55126
02/12/2021, 5:52 PMfaint-motherboard-95438
02/12/2021, 6:04 PMconst domain = 'domain.tld'
const zone = new aws.route53.Zone('main-zone', {
// name: domain,
name: `${domain}.`, // still wants to replace `name` with the exact same value
}, { parent: this })
billowy-army-68599
02/12/2021, 7:06 PMimportant-appointment-55126
02/12/2021, 11:47 PMignoreChanges
didn’t seem to help - it still shows diffs for ~name and ~zoneId when running previewfaint-motherboard-95438
02/16/2021, 8:33 AMbillowy-army-68599
02/16/2021, 7:20 PMignoreChanges
to those fields. I took a look at the diff you sent and it looks like the current state is out of sync with the state of the world. I'm assuming you ran a refresh at some point?gentle-diamond-70147
02/16/2021, 11:13 PMpulumi stack export --file export.json
, then modify the name
property in your Zone's inputs
and outputs
sections to include the trailing period, then pulumi stack import --file export.json
and you should be good to go with your updated code.
Here's an alternative way that achieves the same thing - https://gist.github.com/clstokes/786fb3c0738c1bac6fdd6934992642d2.faint-motherboard-95438
02/17/2021, 9:00 AMrefresh
when I first got the weird diff, since I didn’t change anything, I thought a refresh could help pulumi to get its state back straight, but obviously the trick didn’t work.
@gentle-diamond-70147 thanks for this interesting way of fixing such breaking change. I’m still not used to manipulate the stack state (else than cleaning up pending_operations
when something goes wrong 😬). Would you say that’s currently the “official” way to go to handle such upgrade/breaking changes of pulumi (or any of its related providers) ?
I’ll give it a try tonight (once traffic is down on the platform), I have to get back control on all the records I temporarily added by hand in the zone.
Thanks again for your amazing support & product guys 🙏gentle-diamond-70147
02/18/2021, 5:42 PMimportant-appointment-55126
02/24/2021, 2:16 AMpreview --diff
shows ~ name: "<http://mydomain.com|mydomain.com>." => "<http://mydomain.com|mydomain.com>."
gentle-diamond-70147
02/24/2021, 5:41 AMfaint-motherboard-95438
02/25/2021, 10:58 PMgentle-diamond-70147
02/26/2021, 5:23 AMfaint-motherboard-95438
02/26/2021, 8:40 AM3.23.0
and 3.31.0
?
I just updated the packages, trying again to fix the issue by myself and hoping an upgrade could help, and now it wants me to revert the dot changes 🤪
Before packages upgrade :
+-aws:route53/zone:Zone: (replace)
[id=[REDACTED]]
[urn=urn:pulumi:[REDACTED]$aws:route53/zone:Zone::[REDACTED]-zone]
[provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
~ name: "domain.com." => "domain.com."
After package upgrade
+-aws:route53/zone:Zone: (replace)
[id=[REDACTED]]
[urn=urn:pulumi:[REDACTED]$aws:route53/zone:Zone::[REDACTED]-zone]
[provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_31_0::output<string>]
~ name: "domain.com." => "<http://domain.com|domain.com>"
important-appointment-55126
03/02/2021, 3:04 PMgentle-diamond-70147
03/03/2021, 3:49 AMimportant-appointment-55126
03/03/2021, 4:01 AMfaint-motherboard-95438
03/04/2021, 6:29 PMgentle-diamond-70147
03/04/2021, 6:34 PMfaint-motherboard-95438
03/07/2021, 1:31 PM