What’s the best way to resolve this type of issue:...
# kubernetesb
bored-table-20691
06/09/2021, 1:41 AMWhat’s the best way to resolve this type of issue:
Copy code
error: pre-step event returned an error: failed to verify snapshot: resource urn:pulumi:ssa-us-west-2::okera-infra-regions::kubernetes:yaml:ConfigFile$kubernetes:core/v1:ServiceAccount::cert-manager/cert-manager-webhook refers to unknown provider urn:pulumi:ssa-us-west-2::okera-infra-regions::pulumi:providers:kubernetes::k8s-ssa-provider::460da6b8-808b-4d03-b8f8-ee2fdc9ec693pulumi up -fpulumi refreshbored-table-20691
06/09/2021, 1:43 AMpulumi previewbored-table-20691
06/09/2021, 1:50 AMAfter a preview and then a
pulumi up Copy code
error: the current deployment has 1 resource(s) with pending operations:
* urn:pulumi:ssa-us-west-2::okera-infra-regions::pulumi:providers:kubernetes::k8s-ssa-provider, interrupted while creating
These resources are in an unknown state because the Pulumi CLI was interrupted while
waiting for changes to these resources to complete. You should confirm whether or not the
operations listed completed successfully by checking the state of the appropriate provider.
For example, if you are using AWS, you can confirm using the AWS Console.
Once you have confirmed the status of the interrupted operations, you can repair your stack
using 'pulumi stack export' to export your stack to a file. For each operation that succeeded,
remove that operation from the "pending_operations" section of the file. Once this is complete,
use 'pulumi stack import' to import the repaired stack.
refusing to proceedbored-table-20691
06/09/2021, 1:51 AMIf I export the stack, remove the penidng operation and try to re-import it:
Copy code
$ pulumi stack import --file stack.out
error: 2 errors occurred:
1) state file contains errors: resource urn:pulumi:ssa-us-west-2::okera-infra-regions::kubernetes:yaml:ConfigFile$kubernetes:core/v1:ServiceAccount::cert-manager/cert-manager-webhook refers to unknown provider urn:pulumi:ssa-us-west-2::okera-infra-regions::pulumi:providers:kubernetes::k8s-ssa-provider::460da6b8-808b-4d03-b8f8-ee2fdc9ec693s
steep-toddler-94095
06/09/2021, 2:04 AMweird. do you know what happened right before this issue surfaced?
steep-toddler-94095
06/09/2021, 2:06 AMis any other resource in the stack using that same provider?
b
bored-table-20691
06/09/2021, 2:23 AMI likely switched the provider from one that was constructed manually from a LookupCluster result to one that came form pulumi-EKS
bored-table-20691
06/09/2021, 2:24 AMSo I don’t know if I just got stuff into an unusable state
bored-table-20691
06/09/2021, 2:24 AMAlso one thing I had briefly done was remove the provider role thing on the cluster, and then added it back
bored-table-20691
06/09/2021, 2:24 AMSo maybe it got confused by that
bored-table-20691
06/09/2021, 2:26 AMI’m also ok just destroying the stack and starting over if that’ll work
s
steep-toddler-94095
06/09/2021, 2:28 AMyeah that's that I'd do TBH. in the past i've tried to preserve a resource while adding an existing provider to it by manually editing the statefile and was never able to make it work. I'd be curious what a Pulumi employee has to say about this
b
bored-table-20691
06/09/2021, 2:29 AMWill destroy just work even though the state is messed up?
s
steep-toddler-94095
06/09/2021, 2:31 AMoh right. may have to remove that resource from the pulumi stack, run destroy, and manually delete the resource. in my case everything was in a k8s namespace so I just deleted the namespace and removed the stack, but your situation may not be so easy
b
bored-table-20691
06/09/2021, 2:52 AMIf I destroy the stack it will destroy the EKS cluster too so I guess that’ll solve it. 😀 one issue is I’m worried there’s over a hundred resources in this jammed state so it may be a bit painful
s
steep-toddler-94095
06/09/2021, 3:08 AMactually that was silly of me to suggest. if you can delete only that resource from the state and then manually delete it, you could just then run
pulumi upb
bored-table-20691
06/09/2021, 3:14 AMDumb question - how to delete just that resource?
w
white-balloon-205
06/09/2021, 3:18 AM
@bored-table-20691 the original error you got suggests something got malformed in your state file. It is very unexpected that that could happen via normal use of the Pulumi cli - this most often happens when manual edits have been made to a state file. If it did happen organically - would love to understand the steps that led to that.
But for the issue itself - you should be able to fix up the state file to be correct without destroying any resources. Can you share either the whole state file, or at least the set of provider instances and references (perhaps in DM if that helps?). You most likely just need to ensure that all references to the provider are using a name that exists in the statefile - it should be relatively easy to identify the fix needed by a find/replace on the state.
b
bored-table-20691
06/09/2021, 3:25 AM@white-balloon-205 thanks for replying. It definitely was organic, I don’t modify the state file manually.
I think roughly the steps were:
1. I had my stack as before, which created the kubeconfig (and k8s provider from it) using an
aws/ekspulumi-eksProviderCredentialOptspulumi upProviderCredentialOptspulumi upbored-table-20691
06/09/2021, 5:38 AMOK I edited the state file manually and managed to resurrect things by replacing the bad provider ID with the good one
🧟 1
👍🏾 1
4 Views