This message was deleted.
# generals
sparse-intern-71089
08/01/2019, 5:52 PMThis message was deleted.
w
white-balloon-205
08/01/2019, 6:03 PM
EKS, GKE, AKS? What diff did it report?
h
helpful-advantage-49286
08/01/2019, 6:04 PMGKE
helpful-advantage-49286
08/01/2019, 6:04 PMno diff
helpful-advantage-49286
08/01/2019, 6:04 PMNow everything is pretty hosed because nothing in Pulumi believes it needs to be reinstalled
helpful-advantage-49286
08/01/2019, 6:04 PMso I think I need to clear the state and just rebuild from 0
helpful-advantage-49286
08/01/2019, 6:05 PMwhats the right way to convince a pulumi stack to forget about itself completely so I can go clean it up on the gcp side
helpful-advantage-49286
08/01/2019, 6:05 PMalso, this is kinda nuts to have to do in prod w/ a system that is supposed to prevent this! 😉
w
white-balloon-205
08/01/2019, 6:06 PM
Several parts of what you are describing should not have happened:
1. You should not have seen a replace proposed if you did not make changes
2. If a replace was proposed, you should have seen a diff describing what had changed
3. If a replace was proposed, you should have had the chance to review that diff before accepting
4. If you accepted, it should have succeeded to replace
h
helpful-advantage-49286
08/01/2019, 6:07 PMit wasnt a replace proposed
helpful-advantage-49286
08/01/2019, 6:07 PMit said ‘update’
helpful-advantage-49286
08/01/2019, 6:07 PMand then it went ahead and replaced
helpful-advantage-49286
08/01/2019, 6:07 PMPreviewing update (getmargin/infrastructure):
Type Name Plan Info
pulumipulumiStack infrastructure-infrastructure
~ ├─ gcpcontainerCluster kube update
~ ├─ pulumiproviderskubernetes kube update [diff: ~kubeconfig]
- ├─ gcpdnsRecordSet compoundco-cv-cname delete
- └─ gcpdnsRecordSet getmargin-cv-cname delete
w
white-balloon-205
08/01/2019, 6:08 PM
it said ‘update’Oh - wow - are you sure? That would be extremely bad and my understanding is we have several deep safeguards in place to ensure that cannot happen. If you have any logs of what happened to get there - we would love to see them so we can investigate.
h
helpful-advantage-49286
08/01/2019, 6:08 PMI pasted you the output
helpful-advantage-49286
08/01/2019, 6:08 PMnot sure what logs you need
w
white-balloon-205
08/01/2019, 6:08 PM
Do you have the logs from the replace when it happened?
white-balloon-205
08/01/2019, 6:09 PM
Logs = output from
pulumi updateh
helpful-advantage-49286
08/01/2019, 6:10 PMLogs look normal
helpful-advantage-49286
08/01/2019, 6:10 PMit was just slow
helpful-advantage-49286
08/01/2019, 6:10 PMgot concerned
helpful-advantage-49286
08/01/2019, 6:10 PMsite went down
helpful-advantage-49286
08/01/2019, 6:10 PMchecked cloud console and it said the cluster was being deleted
helpful-advantage-49286
08/01/2019, 6:10 PMit created a new cluster
helpful-advantage-49286
08/01/2019, 6:10 PMbut now that new cluster is pretty f’d
w
white-balloon-205
08/01/2019, 6:12 PM
Logs look normalSpecifically - did your
pulumi updatereplaceupdateh
helpful-advantage-49286
08/01/2019, 6:12 PMyup
w
white-balloon-205
08/01/2019, 6:12 PM
Also - what versions of
@pulumi/gcppulumih
helpful-advantage-49286
08/01/2019, 6:13 PMold
helpful-advantage-49286
08/01/2019, 6:13 PM0.16
helpful-advantage-49286
08/01/2019, 6:13 PMone sec, lemme fix prod real quick the long way
w
white-balloon-205
08/01/2019, 6:13 PM
Ahh - okay. I think this is https://github.com/pulumi/pulumi-terraform/issues/362.
white-balloon-205
08/01/2019, 6:14 PM
That was fixed 3 months ago.
white-balloon-205
08/01/2019, 6:25 PM
Feel free to DM me if you need any help recovering from this and getting the stack back into a working state.
h
helpful-advantage-49286
08/01/2019, 6:35 PMWhats the easiest way to downgrade pulumi to a specific version
w
white-balloon-205
08/01/2019, 6:35 PM
CLI or packages?
h
helpful-advantage-49286
08/01/2019, 6:35 PMCLI
w
white-balloon-205
08/01/2019, 6:37 PM
curl -fsSL <https://get.pulumi.com/> | bash -s -- --version 0.17.26