https://pulumi.com logo
Join Slack
Powered by
How can I get Pulumi to stop trying to delete+crea...
# general
d
How can I get Pulumi to stop trying to delete+create a resource ? I run Pulumi through a script on a VM, so I protect the VM it's currently running on, but after upgrading 
@pulumi/gcp
from 6.x to 7.x, it suddenly wants to delete+create that VM, which fails because it's protected. I already have things like 
ignoreChanges: ['*', '**', '*.*', '**.**']
, but Pulumi still exits in error because it wants to delete the protected resource
l
What reason is it giving for deciding to replace the resource? If it's the same resource, the preview will have a 
~
beside a property that is listed with a recycle icon in the API docs. Changing these properties forces a replacement even if you don't want to: that rule comes from the provider. For example, changing an S3 bucket's name forces a replacement, you cannot change the name of an existing AWS bucket.
If it's two separate entries, one a delete and one a create, then you've changed the resource name (1st parameter). In this case, just undo that change and it should work again.
d
It doesn't give one, and it doesn't indicate it's a replacement, just a create+delete
l
Then you've changed the name
d
The resource name didn't change
Negative
The URN doesn't change
Copy code
-            [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:serviceAccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]

+            [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:serviceaccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]
l
Is the provider changing?
d
Yes
Copy code
-        - gcp:compute/instance:Instance: (delete) 🔒
-            [id=projects/spatial-garage-385109/zones/us-central1-f/instances/vm-deployment-manager-us-central1-47f639f]
-            [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:serviceAccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]
-            [provider=urn:pulumi:staging::keeta-consumer-production-cloud::pulumi:providers:gcp::default_6_54_0::f1fbadba-1b27-4f92-9518-8134e81ecdfe]

+        + gcp:compute/instance:Instance: (create) 🔒
+            [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:serviceaccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]
+            [provider=urn:pulumi:staging::keeta-consumer-production-cloud::pulumi:providers:gcp::default_7_29_0::431e649d-839e-431d-a1fc-51fc289dbe61]
(Because I'm upgrading gcp from 6.x to 7.x -- there are hundreds of provider changes)
l
There you go. New provider. From Pulumi's point of view, you're putting the resource in a whole new cloud.
d
Okay, how can I get it to stop trying to modify this resource ? The change cannot complete, since it'll be deleting itself
l
Change the provider back.
d
Specifying a 
version
didn't help
I don't and can't have both versions of 
@pulumi/gcp
installed at the same time, making it impossible to change the provider back by referencing the package
I really just want it to ignore changes to this resource, since any attempt to delete it will fail, since it'll kill itself in the middle of the process
I have multiple VMs this process can run on, so I can swap it to a different, upgraded one, after this so everything is up-to-date
l
I think I've read something in Slack about ways to avoid this; maybe it was aliases? Not sure. Personally, I'd export the state, edit it to have the new provider, and re-import it. It's a bit nerve-wracking, but if you do it right, then it is a very clean switch.
d
After specifying a 
version
, the provider is no longer changing, but it is still trying to delete+create
Copy code
+ gcp:compute/instance:Instance: (create) 🔒
                [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:compute/network:Network$gcp:compute/subnetwork:Subnetwork$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]
                [provider=urn:pulumi:staging::keeta-consumer-production-cloud::pulumi:providers:gcp::default_6_54_0::f1fbadba-1b27-4f92-9518-8134e81ecdfe]
Copy code
- gcp:compute/instance:Instance: (delete) 🔒
            [id=projects/spatial-garage-385109/zones/us-central1-f/instances/vm-deployment-manager-us-central1-47f639f]
            [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:serviceAccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]
            [provider=urn:pulumi:staging::keeta-consumer-production-cloud::pulumi:providers:gcp::default_6_54_0::f1fbadba-1b27-4f92-9518-8134e81ecdfe]
l
Does your state have a pending delete for it? There's a section at the bottom of the state file for recording pending actions. Maybe that needs to be cleaned too.
d
(The URN changed there because I tried removing "parent", let me capture the delete+create with the parent still present)
l
There's a way to clean pending actions via the CLI...
I see 
pulumi refresh --clear-pending-creates
. Not sure if that's what you want...
d
Nothing pending deletion as far as I can see:
Copy code
$ pulumi stack export | jq --tab '. | keys'
[
	"deployment",
	"version"
]
$ pulumi stack export | jq --tab '.deployment | keys'
[
	"manifest",
	"resources",
	"secrets_providers"
]
$
l
I think it's one more level down, inside the "resources" property
(BTW that's a handy jq syntax, I've always relied on vscode for JSON searching.. I should learn more jq)
d
Resources is an array, so I don't think that's it
Copy code
$ pulumi stack export | jq --tab '.deployment.resources | keys' | wc -l
738
$ pulumi stack export | jq --tab '.deployment.resources | keys' | tail
	727,
	728,
	729,
	730,
	731,
	732,
	733,
	734,
	735
]
l
It's been ages since I looked inside the state, but if I recall correctly, there are fake resource entries for pending actions. Maybe this is a time when browsing the JSON might be good 🙂
d
From what I recall there was a top-level or second-level 'delete' key which held resources scheduled for deletion
I'm pretty sure the current state doesn't expect any deletions
Nowhere in the current state is there any key named `delete`:
Copy code
$ pulumi stack export | jq --tab '.' | grep -3 '"delete"'
$
l
Ah. Then I don't know why it's still expecting to replace the resource. You've fixed the provider. There's no helpful hint in the preview?
d
Nope, to even get anything helpful I'm using 
--diff
The parent's provider is still changing, though it's not being deleted+created or replaced, just the provider is updating
(And its parent has no parent, so the parent of it is just the stack itself)
Interestingly, change its parents version back to 6.x causes a different failure:
Copy code
~ pulumi:providers:gcp: (update)
            [id=8f1d653f-2285-46c8-bef4-3ddbeede25fe]
            [urn=urn:pulumi:staging::keeta-consumer-production-cloud::keeta:gcp:Deployment$pulumi:providers:gcp::keetanet-cluster-base-provider]
          ~ version: "6.54.0" => "7.29.0"

error: unrecognized resource type (Check): gcp:serviceaccount/account:Account
Interestingly, the preview fails even if I set "`protect: false`" because it thinks it is protected (I guess that's what the CURRENT state says, though the new state won't)
Oh, the URN of the resource is slightly different, the case of 
serviceAccount
became all lower-cased
Copy code
NEW: [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:service**a**ccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]

OLD: [urn=urn:pulumi:staging::keeta-consumer-production-cloud::gcp:service**A**ccount/account:Account$gcp:compute/instance:Instance::vm-deployment-manager-us-central1]
Adding an alias fixed it -- but after it was deployed the stack is now pretty much hosed due to other bugs
l
Very unfortunate. I have found that Pulumi-protected resources can cause a lot of trouble. For resources that have provider-protected options (e.g. RDS databases), I prefer that option.
d
Well, the protect flag was working as intended, it was the replacement occurring due to changing the parent resource name's case that broke it -- I can just replace the backup VMs first and leave the existing one alone with an alias, and then swap it and it'll fix it... but now due to bugs in ManagedZone the stack can't be parsed
👍 1
Copy code
error: Error: Invalid asset encountered when unmarshaling resource property
        at deserializeProperty (/home/rkeene/devel/keeta-production/node_modules/@pulumi/runtime/rpc.ts:578:27)
        at Object.deserializeProperties (/home/rkeene/devel/keeta-production/node_modules/@pulumi/runtime/rpc.ts:181:24)
        at /home/rkeene/devel/keeta-production/node_modules/@pulumi/runtime/resource.ts:798:33
Originally it started out with this error: https://github.com/pulumi/pulumi-gcp/issues/1036 (which is supposedly fixed, but apparently not actually)
I don't know if I can even rollback from this... I guess I have the old state files, since they're versioned... some resources will be a bit weird
(fortunately this isn't production, yet)
292 Views
Open in Slack
PreviousNext
Powered by