Hi there,

pulumi-aws

just trashed all my main zone without any reason (in production, else that’s not fun), while i didn’t change anything for ages related to this zone or its records (i was updating something else in the stack):

+-aws:route53/zone:Zone: (replace)
            [id=[REDACTED]]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "[REDACTED]." => "[REDACTED]"
        ++aws:route53/zone:Zone: (create-replacement)
            [id=[REDACTED]]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "[REDACTED]." => "[REDACTED]"
        +-aws:route53/record:Record: (replace)
            [id=[REDACTED]_TXT]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/record:Record::[REDACTED]-record-3-TXT]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ zoneId : "[REDACTED]" => output<string>

Since most of the time it just wants to update some props related to zone or records (that’s weird btw), I didn’t pay attention (my fault), but this time it trashed all my records, trying to replace them without any reason nor changes, and failed when it tried to delete the main zone (obviously since it’s not empty). Now it’s stuck deleting the records & the zone and preventing me to recreate them ! I had to quickly recreate everything by hand and now I have a stack completely stale, desync and unusable… I suppose that’s related to the update of

pulumi-aws

, but even trying to downgrade to

2.x

it still wants to delete the zone by renaming it

zone.tld.

=>

zone.tld

(which does not make sense to me since I didn’t put a trailing dot in the domain name in the first place nor change anything in that regard) That’s impossible to maintain a production stack when it wants to delete / replace something as critical as your main NS zone, I don’t see the point here and it does not seem there’s any way to prevent it. What should I have done and how can I save the stack now (without deleting the zone) ?

Hey, I'm sorry this happened, it sounds like you upgraded from

2.x

to

3.x

? can you share the code you currently have? I think just adding a trailing dot would be enough to fix this, but not entirely sure

hey @billowy-army-68599, thanks for answering. I’ve updated a long time ago to

3.x

but we nearly never update the stack handling our main NS zone so that’s why I encountered this only today. I can try to share the code but it’s pretty basic, one

aws.route53.Zone

followed by a few

aws.route53.Records

, nothing special. As you can see the main problem is that the version change force a delete+replace on all

aws.route53

resources. That’s basically impossible to even think about it in a real life production context, that would mean being stuck in

2.x

for life for a given production environment.

I don't think it's necessary to have to stay on

2.x

- I believe you just need to update your

Zone

resource to add the trailing dot or alias the resource.

Nope, still wants to replace it because of

provider

and

name

(giving that now

name

is the same between old and new value, with a trailing dot, don’t know why it still wants to change it) :

+-aws:route53/zone:Zone: (replace)
            [id=[REDACTED]]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "[REDACTED]." => "[REDACTED]."

i’m having a similar issue - i thought i’d worked around it by adding the trailing period it wants to the zone name, but it still seems to want to do a zone replacement due to a change in

delegationSetId

@important-appointment-55126 can you use

ignoreChanges

for that field?

good idea; will try that

I can’t see how to avoid a replacement without changing anything in the module source code, given how basic the implementation is :

const domain = 'domain.tld'

const zone = new aws.route53.Zone('main-zone', {
  // name: domain,
  name: `${domain}.`, // still wants to replace `name` with the exact same value
}, { parent: this })

can you share your diff?

ignoreChanges

didn’t seem to help - it still shows diffs for ~name and ~zoneId when running preview

@billowy-army-68599 I sent you the diff in PM yesterday. No rush, but let me know what you think and if I should open an issue ? Thanks

hey, been mulling this over during the weekend. I'm not totally sure how to recommend you proceed due to the breaking changed (v2 -> v3) upgrade. Generally I'd recommend adding

ignoreChanges

to those fields. I took a look at the diff you sent and it looks like the current state is out of sync with the state of the world. I'm assuming you ran a refresh at some point?

@faint-motherboard-95438 I'm sorry it took me awhile to see this thread. Another customer of ours ran into this awhile back. IIRC, the name without the trailing period is still in your state file, hence why Pulumi thinks it needs to replace the Zone. So you can

pulumi stack export --file export.json

, then modify the

name

property in your Zone's

inputs

and

outputs

sections to include the trailing period, then

pulumi stack import --file export.json

and you should be good to go with your updated code. Here's an alternative way that achieves the same thing - https://gist.github.com/clstokes/786fb3c0738c1bac6fdd6934992642d2.

Hey @billowy-army-68599 & @gentle-diamond-70147 thanks for getting back to me ! @billowy-army-68599 IIRC i did a

refresh

when I first got the weird diff, since I didn’t change anything, I thought a refresh could help pulumi to get its state back straight, but obviously the trick didn’t work. @gentle-diamond-70147 thanks for this interesting way of fixing such breaking change. I’m still not used to manipulate the stack state (else than cleaning up

pending_operations

when something goes wrong 😬). Would you say that’s currently the “official” way to go to handle such upgrade/breaking changes of pulumi (or any of its related providers) ? I’ll give it a try tonight (once traffic is down on the platform), I have to get back control on all the records I temporarily added by hand in the zone. Thanks again for your amazing support & product guys 🙏

This is our "officially recommended way" to fix this particular issue. 🙂 In general, you should not need to do state manipulation or extra steps like this for provider upgrades. Unfortunately, this is a rare exception and an edge case that needs an extra step.

i'm still missing something here.. updated the state file, added a period to the input/output names for the zone and updated my code to include the trailing period for the resource definition and

preview --diff

shows

~ name: "<http://mydomain.com|mydomain.com>." => "<http://mydomain.com|mydomain.com>."

@important-appointment-55126 are you seeing that on the Zone or on Record resources? Would you be able to share your state file with me (via DM)?

Hi @gentle-diamond-70147, sorry for the late reply but I had other priorities this week to attend to and I didn’t give a try to your fix. I can confirm that, like @important-appointment-55126 , both manually editing the state file or following your gist does not fix the problem and the diff still wants to replace the zone with the exact same name.

Shoot. Thanks for trying. It's been a while since I worked through this so give me some time to reproduce this situation and get you a better answer. Thanks for your patience.

@gentle-diamond-70147 sure, no worries, that’s not critical, just inconvenient not been able to manage the zone with pulumi. Since it is isolated in a core stack, and nearly never change, that does not impact our day to day deployments. Let me know if I can help in the process.

that seems odd/concerning.. i haven’t done anything further with my stack while waiting for an update from @gentle-diamond-70147 but it is blocking me making some zone updates 😕

@faint-motherboard-95438 I'm not aware of any changes that would cause this to revert, but I'll do a little digging. @important-appointment-55126 apologies for the delay!

np; appreciate you checking in

@gentle-diamond-70147 really weird then, but I’m pretty sure I got unstuck by the package upgrade and the diff is explicit on reverting the changes it wanted to apply in the first place. Don’t know if I can help figuring this out (nor if that’s really useful ?)

Not sure I follow. Are you back to a good "state"?

@gentle-diamond-70147 yes sorry, I meant helping figuring out what happened (also since @important-appointment-55126 seems to still have the problem), but my state is now consistent, not more issue on this matter.