https://pulumi.com logo
Powered by
h

helpful-advantage-49286

08/01/2019, 5:52 PM
it said it would update my k8s cluster
w

white-balloon-205

08/01/2019, 6:03 PM
EKS, GKE, AKS? What diff did it report?
h

helpful-advantage-49286

08/01/2019, 6:04 PM
GKE
no diff
Now everything is pretty hosed because nothing in Pulumi believes it needs to be reinstalled
so I think I need to clear the state and just rebuild from 0
whats the right way to convince a pulumi stack to forget about itself completely so I can go clean it up on the gcp side
also, this is kinda nuts to have to do in prod w/ a system that is supposed to prevent this! 😉
w

white-balloon-205

08/01/2019, 6:06 PM
Several parts of what you are describing should not have happened: 1. You should not have seen a replace proposed if you did not make changes 2. If a replace was proposed, you should have seen a diff describing what had changed 3. If a replace was proposed, you should have had the chance to review that diff before accepting 4. If you accepted, it should have succeeded to replace
h

helpful-advantage-49286

08/01/2019, 6:07 PM
it wasnt a replace proposed
it said ‘update’
and then it went ahead and replaced
Previewing update (getmargin/infrastructure): Type Name Plan Info pulumipulumiStack infrastructure-infrastructure ~ ├─ gcpcontainerCluster kube update ~ ├─ pulumiproviderskubernetes kube update [diff: ~kubeconfig] - ├─ gcpdnsRecordSet compoundco-cv-cname delete - └─ gcpdnsRecordSet getmargin-cv-cname delete
w

white-balloon-205

08/01/2019, 6:08 PM
it said ‘update’
Oh - wow - are you sure? That would be extremely bad and my understanding is we have several deep safeguards in place to ensure that cannot happen. If you have any logs of what happened to get there - we would love to see them so we can investigate.
h

helpful-advantage-49286

08/01/2019, 6:08 PM
I pasted you the output
not sure what logs you need
w

white-balloon-205

08/01/2019, 6:08 PM
Do you have the logs from the replace when it happened?
Logs = output from 
pulumi update
.
h

helpful-advantage-49286

08/01/2019, 6:10 PM
Logs look normal
it was just slow
got concerned
site went down
checked cloud console and it said the cluster was being deleted
it created a new cluster
but now that new cluster is pretty f’d
w

white-balloon-205

08/01/2019, 6:12 PM
Logs look normal
Specifically - did your 
pulumi update
ever say the word 
replace
on the cluster, or did it also say 
update
but then behind the scenes your cluster was getting deleted?
h

helpful-advantage-49286

08/01/2019, 6:12 PM
yup
w

white-balloon-205

08/01/2019, 6:12 PM
Also - what versions of 
@pulumi/gcp
and the 
pulumi
CLI do you have?
h

helpful-advantage-49286

08/01/2019, 6:13 PM
old
0.16
one sec, lemme fix prod real quick the long way
w

white-balloon-205

08/01/2019, 6:13 PM
That was fixed 3 months ago.
Feel free to DM me if you need any help recovering from this and getting the stack back into a working state.
h

helpful-advantage-49286

08/01/2019, 6:35 PM
Whats the easiest way to downgrade pulumi to a specific version
w

white-balloon-205

08/01/2019, 6:35 PM
CLI or packages?
h

helpful-advantage-49286

08/01/2019, 6:35 PM
CLI
w

white-balloon-205

08/01/2019, 6:37 PM
curl -fsSL <https://get.pulumi.com/> | bash -s -- --version 0.17.26
Powered by