https://pulumi.com logo
Title
f

faint-motherboard-95438

02/12/2021, 10:58 AM
Hi there,
pulumi-aws
just trashed all my main zone without any reason (in production, else that’s not fun), while i didn’t change anything for ages related to this zone or its records (i was updating something else in the stack):
+-aws:route53/zone:Zone: (replace)
            [id=[REDACTED]]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "[REDACTED]." => "[REDACTED]"
        ++aws:route53/zone:Zone: (create-replacement)
            [id=[REDACTED]]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "[REDACTED]." => "[REDACTED]"
        +-aws:route53/record:Record: (replace)
            [id=[REDACTED]_TXT]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/record:Record::[REDACTED]-record-3-TXT]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ zoneId : "[REDACTED]" => output<string>
Since most of the time it just wants to update some props related to zone or records (that’s weird btw), I didn’t pay attention (my fault), but this time it trashed all my records, trying to replace them without any reason nor changes, and failed when it tried to delete the main zone (obviously since it’s not empty). Now it’s stuck deleting the records & the zone and preventing me to recreate them ! I had to quickly recreate everything by hand and now I have a stack completely stale, desync and unusable… I suppose that’s related to the update of
pulumi-aws
, but even trying to downgrade to
2.x
it still wants to delete the zone by renaming it
zone.tld.
=>
zone.tld
(which does not make sense to me since I didn’t put a trailing dot in the domain name in the first place nor change anything in that regard) That’s impossible to maintain a production stack when it wants to delete / replace something as critical as your main NS zone, I don’t see the point here and it does not seem there’s any way to prevent it. What should I have done and how can I save the stack now (without deleting the zone) ?
Seems related to this terraform PR: https://github.com/hashicorp/terraform-provider-aws/pull/14220 Not sure if the final behavior I encountered is specific to pulumi (force delete + recreate of a critical resource) or if I should escalate the problem directly to terraform ? (Since I can’t really try to reproduce with native terraform code)
b

billowy-army-68599

02/12/2021, 4:49 PM
Hey, I'm sorry this happened, it sounds like you upgraded from
2.x
to
3.x
? can you share the code you currently have? I think just adding a trailing dot would be enough to fix this, but not entirely sure
f

faint-motherboard-95438

02/12/2021, 4:54 PM
hey @billowy-army-68599, thanks for answering. I’ve updated a long time ago to
3.x
but we nearly never update the stack handling our main NS zone so that’s why I encountered this only today. I can try to share the code but it’s pretty basic, one
aws.route53.Zone
followed by a few
aws.route53.Records
, nothing special. As you can see the main problem is that the version change force a delete+replace on all
aws.route53
resources. That’s basically impossible to even think about it in a real life production context, that would mean being stuck in
2.x
for life for a given production environment.
apart from the dot issue on the main zone, it also wants to update the provider urn on all resources, but that should not trigger a deletion anyway.
b

billowy-army-68599

02/12/2021, 4:58 PM
I don't think it's necessary to have to stay on
2.x
- I believe you just need to update your
Zone
resource to add the trailing dot or alias the resource.
f

faint-motherboard-95438

02/12/2021, 5:06 PM
Nope, still wants to replace it because of
provider
and
name
(giving that now
name
is the same between old and new value, with a trailing dot, don’t know why it still wants to change it) :
+-aws:route53/zone:Zone: (replace)
            [id=[REDACTED]]
            [urn=urn:pulumi:main::core::[REDACTED]:aws:Zone$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "[REDACTED]." => "[REDACTED]."
Also, about the downgrade to
2.x
, I updated
package.json
, cleaned up
node_modules
&
package-lock.json
, removed all plugins with
pulumi plugins rm resource aws
then did a clean
npm i
and ensured only
pulumi-aws
2.13
is installed, but as soon as I
pulumi up
, it installs the
3.23
plugin before the preview, don’t know why
i

important-appointment-55126

02/12/2021, 5:51 PM
i’m having a similar issue - i thought i’d worked around it by adding the trailing period it wants to the zone name, but it still seems to want to do a zone replacement due to a change in
delegationSetId
b

billowy-army-68599

02/12/2021, 5:52 PM
@important-appointment-55126 can you use
ignoreChanges
for that field?
i

important-appointment-55126

02/12/2021, 5:52 PM
good idea; will try that
f

faint-motherboard-95438

02/12/2021, 6:04 PM
I can’t see how to avoid a replacement without changing anything in the module source code, given how basic the implementation is :
const domain = 'domain.tld'

const zone = new aws.route53.Zone('main-zone', {
  // name: domain,
  name: `${domain}.`, // still wants to replace `name` with the exact same value
}, { parent: this })
b

billowy-army-68599

02/12/2021, 7:06 PM
can you share your diff?
i

important-appointment-55126

02/12/2021, 11:47 PM
ignoreChanges
didn’t seem to help - it still shows diffs for ~name and ~zoneId when running preview
f

faint-motherboard-95438

02/16/2021, 8:33 AM
@billowy-army-68599 I sent you the diff in PM yesterday. No rush, but let me know what you think and if I should open an issue ? Thanks
b

billowy-army-68599

02/16/2021, 7:20 PM
hey, been mulling this over during the weekend. I'm not totally sure how to recommend you proceed due to the breaking changed (v2 -> v3) upgrade. Generally I'd recommend adding
ignoreChanges
to those fields. I took a look at the diff you sent and it looks like the current state is out of sync with the state of the world. I'm assuming you ran a refresh at some point?
g

gentle-diamond-70147

02/16/2021, 11:13 PM
@faint-motherboard-95438 I'm sorry it took me awhile to see this thread. Another customer of ours ran into this awhile back. IIRC, the name without the trailing period is still in your state file, hence why Pulumi thinks it needs to replace the Zone. So you can
pulumi stack export --file export.json
, then modify the
name
property in your Zone's
inputs
and
outputs
sections to include the trailing period, then
pulumi stack import --file export.json
and you should be good to go with your updated code. Here's an alternative way that achieves the same thing - https://gist.github.com/clstokes/786fb3c0738c1bac6fdd6934992642d2.
f

faint-motherboard-95438

02/17/2021, 9:00 AM
Hey @billowy-army-68599 & @gentle-diamond-70147 thanks for getting back to me ! @billowy-army-68599 IIRC i did a
refresh
when I first got the weird diff, since I didn’t change anything, I thought a refresh could help pulumi to get its state back straight, but obviously the trick didn’t work. @gentle-diamond-70147 thanks for this interesting way of fixing such breaking change. I’m still not used to manipulate the stack state (else than cleaning up
pending_operations
when something goes wrong 😬). Would you say that’s currently the “official” way to go to handle such upgrade/breaking changes of pulumi (or any of its related providers) ? I’ll give it a try tonight (once traffic is down on the platform), I have to get back control on all the records I temporarily added by hand in the zone. Thanks again for your amazing support & product guys 🙏
g

gentle-diamond-70147

02/18/2021, 5:42 PM
This is our "officially recommended way" to fix this particular issue. 🙂 In general, you should not need to do state manipulation or extra steps like this for provider upgrades. Unfortunately, this is a rare exception and an edge case that needs an extra step.
i

important-appointment-55126

02/24/2021, 2:16 AM
i'm still missing something here.. updated the state file, added a period to the input/output names for the zone and updated my code to include the trailing period for the resource definition and
preview --diff
shows
~ name: "<http://mydomain.com|mydomain.com>." => "<http://mydomain.com|mydomain.com>."
g

gentle-diamond-70147

02/24/2021, 5:41 AM
@important-appointment-55126 are you seeing that on the Zone or on Record resources? Would you be able to share your state file with me (via DM)?
f

faint-motherboard-95438

02/25/2021, 10:58 PM
Hi @gentle-diamond-70147, sorry for the late reply but I had other priorities this week to attend to and I didn’t give a try to your fix. I can confirm that, like @important-appointment-55126 , both manually editing the state file or following your gist does not fix the problem and the diff still wants to replace the zone with the exact same name.
g

gentle-diamond-70147

02/26/2021, 5:23 AM
Shoot. Thanks for trying. It's been a while since I worked through this so give me some time to reproduce this situation and get you a better answer. Thanks for your patience.
f

faint-motherboard-95438

02/26/2021, 8:40 AM
@gentle-diamond-70147 sure, no worries, that’s not critical, just inconvenient not been able to manage the zone with pulumi. Since it is isolated in a core stack, and nearly never change, that does not impact our day to day deployments. Let me know if I can help in the process.
@gentle-diamond-70147 have you fixed/reverted something in the last releases between the
3.23.0
and
3.31.0
? I just updated the packages, trying again to fix the issue by myself and hoping an upgrade could help, and now it wants me to revert the dot changes 🤪 Before packages upgrade :
+-aws:route53/zone:Zone: (replace)
            [id=[REDACTED]]
            [urn=urn:pulumi:[REDACTED]$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_23_0::0800a252-acfa-4069-ad32-aa0ee18ebb0e]
          ~ name: "domain.com." => "domain.com."
After package upgrade
+-aws:route53/zone:Zone: (replace)
            [id=[REDACTED]]
            [urn=urn:pulumi:[REDACTED]$aws:route53/zone:Zone::[REDACTED]-zone]
            [provider: urn:pulumi:main::core::pulumi:providers:aws::default_2_13_1::d1d769bc-62a1-45ba-b339-e706b31a6fbd => urn:pulumi:main::core::pulumi:providers:aws::default_3_31_0::output<string>]
          ~ name: "domain.com." => "<http://domain.com|domain.com>"
anyway, the good news is, now I can fix the stack.. by reverting the changes it wanted me to apply in the first place
i

important-appointment-55126

03/02/2021, 3:04 PM
that seems odd/concerning.. i haven’t done anything further with my stack while waiting for an update from @gentle-diamond-70147 but it is blocking me making some zone updates 😕
g

gentle-diamond-70147

03/03/2021, 3:49 AM
@faint-motherboard-95438 I'm not aware of any changes that would cause this to revert, but I'll do a little digging. @important-appointment-55126 apologies for the delay!
i

important-appointment-55126

03/03/2021, 4:01 AM
np; appreciate you checking in
f

faint-motherboard-95438

03/04/2021, 6:29 PM
@gentle-diamond-70147 really weird then, but I’m pretty sure I got unstuck by the package upgrade and the diff is explicit on reverting the changes it wanted to apply in the first place. Don’t know if I can help figuring this out (nor if that’s really useful ?)
g

gentle-diamond-70147

03/04/2021, 6:34 PM
Not sure I follow. Are you back to a good "state"?
f

faint-motherboard-95438

03/07/2021, 1:31 PM
@gentle-diamond-70147 yes sorry, I meant helping figuring out what happened (also since @important-appointment-55126 seems to still have the problem), but my state is now consistent, not more issue on this matter.