https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
b
Hi we've got some instructions on how you can deal with pending operations: https://www.pulumi.com/docs/troubleshooting/#interrupted-update-recovery
b
hi @brave-planet-10645, thanks for the response. I did try this. We're using an S3 pulumi backend. The 
pulumi cancel
command fails I think because we're using S3? But I did the other steps to no effect.
b
Were you not able to export the state file?
b
I am able to export the state
pulumi cancel
says "error: the 
cancel
command is not supported for local stacks" when im logged into my S3 backend
b
Ok, leave the pulumi cancel for a moment. In the state file you've exported, can you find the 
pending_operations
section?
Actually, ignore that as well for a moment. What happens when you run
Copy code
pulumi stack export | pulumi stack import
?
b
it says the import completed successfully
b
But then when you run 
pulumi refresh
that fails?
b
negative, 
pulumi refresh
succeeds
b
and then 
pulumi destroy
fails?
b
yes it tries to start by destroying the ACM cert saying "completing deletion from previous update"
which will fail after 20 minutes because "certificate is in use" because it needs to delete the ALB / HTTPS listener first
b
oh so you're not specifically asking about the pending operations within Pulumi? Sorry I misunderstood. I'm assuming you can't delete the ALB or the listener because it says that you need to delete the certificate first?
b
well I guess it's 2 issues - 1. pulumi keeps acting like it is resuming a previous update by deleting the cert first thing after starting a destroy and 2. pulumi keeps trying to delete the cert before the ALB / HTTPS listener
b
So if you export your stack and then in the file you export it to search for 
pending_operations
(there should be one about deleting the certificate) and remove that. Then run 
pulumi stack import --file /path/to/file
to re-import the stack
then run pulumi destroy again and it should skip over the cert
b
hmmm well the exported file doesn't contain the word "pending" anywhere
b
Could you please run the following command 
pulumi destroy --logtostderr -v=9 2> out.txt
and then attach the out,txt file you've just created and also the stack file you've exported to an email and send it to support@pulumi.com so I can take a look
b
can do, give me 20 minutes to run the destroy
b
thank you
b
files sent. Thank you for looking into this
b
No problem. I'll look at the email and continue the conversation on there
A couple of things to try: what happens if you run 
pulumi refresh
?
b
nothing unusual, it only finds changes in the RDS instance each time. but it succeeds
b
can you then run pulumi up and then pulumi destroy... from looking around, I can see that this is a common issue with terraform as well so trying to dig into how to fix your pulumi CLI issue first and then we can look at deleting things
b
unfortunately a 
pulumi up
also starts by trying to delete the cert, and thus fails
b
have you tried 
pulumi stack export | pulumi stack import
?
Also, if you go into the console, can you see the certificate in ACM?
b
yea we've tried that it doesn't seem to have any impact. and yes the console contains 2 certs, I believe pulumi is failing to delete the old one after creating the new one
b
I think I've found the issue, but I need to check something with a colleague first
b
sure, and thanks again!
b
right, so in your export (might be worth grabbing another in case you've made changes in the meantime) one of the records for the cert that's being deleted has an extra field called "delete" and it's set to true. If you delete that field and then import that file to your stack it will stop the 
completing deletion...
from happening
I would do this by running the export, taking a copy of that file and then removing that field
I've got it on line 
2631
- if you want to send me a fresh export, I don't mind removing it then sending the file back
b
Sorry Piers had a meeting. So I just tried that and on the import step I get:
Copy code
error: 2 errors occurred:
    1) state file contains errors: duplicate resource urn:pulumi:qa::fig-grafana::aws:acm/certificate:Certificate::ssl-cert (not marked for deletion)
    2) importing this file could be dangerous; rerun with --force to proceed anyway
b
ok so let's not do this right now
Let me think and I'll get back to you
b
alright then, thank you
b
To recap on this, are you trying to tear down the whole infrastructure here or just delete the certificate?
b
It is a stack that I am actively working on - I added some additional SANs to the cert which triggered a replace action on the cert which failed with that issue. At some point I just started trying to blow it all away to start from scratch. I needed to make progress yesterday so I ended up manually transferring the listener to the new cert and deleting the old one. Then I did a full destroy which I was able to do successfully, which included deleting the new (now only) cert. So I think it was specifically an issue with the replace action. Once I have this stack to where I want it I'm going to see If I can reproduce the error with this stack before I blow it away.
b
If it helps, I've got a basic set up with one of my personal domains here: https://github.com/pierskarsenbarg/pulumi-scratchpad/blob/main/cert-alb/index.ts
I was able to spin that up and destroy it with no issues. I'll try it with adding something to the cert and see if that triggers the same problem.
b
that looks like it would work. Yea the issue started when the cert already existed and I added subject alternates. So you'll also need more domain validation records too
also I want to note that in our stack our listener depends on 
validation.CertificateArn
and not the cert itself
b
Really? What does that give you over the original cert arn (asking for educational reasons, not that I think it's the wrong thing to do)
b
so the 
cert
being deployed doesn't necessarily mean that it can be used yet, because 
cert
can be considered deployed before the certificate has passed domain validation. But by depending on the 
validation
we ensure that the cert isn't used until it is actually validated
b
that makes absolute sense. That's really smart. I'm making a note of that for the future
b
Here is that segment of our code:
Untitled
11 Views
Open in Slack
PreviousNext
Powered by