https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
m
pulumi refresh
warns the following:
Copy code
warning: Refreshed resource is in an unhealthy state:
    * Resource 'compiler-deployment' was created but failed to initialize
    * [ProgressDeadlineExceeded] ReplicaSet "compiler-deployment-7f9fbbdf4f" has timed out progressing.
    * Minimum number of Pods to consider the application live was not attained
oh well… i just destroyed the stack and re-created it… problem seems to have gone away now. but im really scared it’ll happen again if i try to make another failed deployment :(
g
Can you share the full log you got from the 
up
? I suspect there's another error message in the output that will give us more information.
m
So… this problem came back again today 😢 let me see if I can share the full log from 
pulumi up
Some interesting output, but not sure if root cause:
Copy code
I1012 21:26:46.600447     642 plugins.go:100] gatherPluginsFromSnapshot(): gathering plugins from snapshot
I1012 21:26:46.600524     642 plugins.go:117] gatherPluginsFromSnapshot(): plugin aws 3.2.1 is required by first-class provider "urn:pulumi:dev::services::pulumi:providers:aws::default_3_2_1"
I1012 21:26:46.600994     642 plugins.go:117] gatherPluginsFromSnapshot(): plugin pulumi <nil> is required by first-class provider "urn:pulumi:dev::services::pulumi:providers:pulumi::default"
I1012 21:26:46.601131     642 plugins.go:117] gatherPluginsFromSnapshot(): plugin kubernetes <nil> is required by first-class provider "urn:pulumi:dev::services::pulumi:providers:kubernetes::infra-cluster-provider"
I1012 21:26:46.601407     642 plugins.go:132] ensurePluginsAreInstalled(): beginning
I1012 21:26:46.601550     642 plugins.go:461] GetPluginPath(language, nodejs, <nil>): found on $PATH /usr/local/bin/pulumi-language-nodejs
I1012 21:26:46.601641     642 plugins.go:137] ensurePluginsAreInstalled(): plugin nodejs <nil> already installed
I1012 21:26:46.602606     642 plugins.go:500] GetPluginPath(resource, aws, 3.2.1): enabling new plugin behavior
I1012 21:26:46.602666     642 plugins.go:582] SelectCompatiblePlugin(..., aws): beginning
I1012 21:26:46.602696     642 plugins.go:611] SelectCompatiblePlugin(..., aws): best plugin aws-3.2.1: semver match
I1012 21:26:46.602713     642 plugins.go:624] SelectCompatiblePlugin(..., aws): selecting plugin 'aws-3.2.1': best match
I1012 21:26:46.602765     642 plugins.go:547] GetPluginPath(resource, aws, 3.2.1): found in cache at /home/toast/.pulumi/plugins/resource-aws-v3.2.1/pulumi-resource-aws
I1012 21:26:46.602819     642 plugins.go:137] ensurePluginsAreInstalled(): plugin aws 3.2.1 already installed
I1012 21:26:46.604009     642 plugins.go:145] ensurePluginsAreInstalled(): plugin pulumi <nil> not installed, doing install
I1012 21:26:46.604128     642 plugins.go:164] installPlugin(pulumi, <nil>): beginning install
I1012 21:26:46.604446     642 plugins.go:530] GetPluginPath(resource, kubernetes, <nil>): found candidate (#2.6.0)
I1012 21:26:46.604507     642 plugins.go:547] GetPluginPath(resource, kubernetes, <nil>): found in cache at /home/toast/.pulumi/plugins/resource-kubernetes-v2.6.0/pulumi-resource-kubernetes
I1012 21:26:46.604589     642 plugins.go:137] ensurePluginsAreInstalled(): plugin kubernetes <nil> already installed
I1012 21:26:46.785635     642 plugins.go:152] ensurePluginsAreInstalled(): completed
I1012 21:26:46.785731     642 update.go:229] newUpdateSource(): failed to install missing plugins: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-pulumi-v%3Cnil%3E-linux-amd64.tar.gz>
I1012 21:26:46.785818     642 plugins.go:220] computeDefaultProviderPlugins(): language host reported empty set of provider plugins, using all plugins
I1012 21:26:46.786048     642 plugins.go:239] computeDefaultProviderPlugins(): considering nodejs
I1012 21:26:46.786331     642 plugins.go:239] computeDefaultProviderPlugins(): considering pulumi
I1012 21:26:46.786366     642 plugins.go:270] computeDefaultProviderPlugins(): plugin pulumi selected for package pulumi (first seen)
I1012 21:26:46.786614     642 plugins.go:239] computeDefaultProviderPlugins(): considering kubernetes
I1012 21:26:46.786648     642 plugins.go:270] computeDefaultProviderPlugins(): plugin kubernetes selected for package kubernetes (first seen)
I1012 21:26:46.786677     642 plugins.go:239] computeDefaultProviderPlugins(): considering aws-3.2.1
I1012 21:26:46.786774     642 plugins.go:270] computeDefaultProviderPlugins(): plugin aws-3.2.1 selected for package aws (first seen)
I1012 21:26:46.786810     642 plugins.go:276] computeDefaultProviderPlugins(): summary of default plugins:
I1012 21:26:46.786865     642 plugins.go:278]   pulumi          = <nil>
I1012 21:26:46.787057     642 plugins.go:278]   kubernetes      = <nil>
I1012 21:26:46.787729     642 plugins.go:278]   aws             = 3.2.1
It’s weird that it tried to download 
<https://get.pulumi.com/releases/plugins/pulumi-resource-pulumi-v%3Cnil%3E-linux-amd64.tar.gz>
the actual failure:
Copy code
I1012 21:26:56.512666     642 source_eval.go:158] EvalSourceIterator produced a registration: t=kubernetes:core/v1:Namespace,name=jetpack,#props=3
I1012 21:26:56.513188     642 plan_executor.go:219] planExecutor.Execute(...): incoming event (nil? false, <nil>)
I1012 21:26:56.513353     642 plan_executor.go:364] planExecutor.handleSingleEvent(...): received RegisterResourceEvent
I1012 21:26:56.513478     642 registry.go:146] GetProvider(urn:pulumi:dev::services::pulumi:providers:kubernetes::infra-cluster-provider::624d6186-2b3e-4ef3-ad02-7847fb4f14d6)
I1012 21:26:56.513754     642 provider_plugin.go:488] Provider[kubernetes, 0xc000b2b4a0].Check(urn:pulumi:dev::services::kubernetes:core/v1:Namespace::jetpack) executing (#olds=3,#news=3
I1012 21:26:56.514106     642 plan_executor.go:237] planExecutor.Execute(...): error handling event: transport is closing
I1012 21:26:56.514342     642 eventsink.go:86] eventSink::Error(<{%reset%}>transport is closing<{%reset%}>)
I1012 21:26:56.514557     642 step_executor.go:321] StepExecutor worker(-1): StepExecutor.waitForCompletion(): waiting for worker threads to exit
I1012 21:26:56.514808     642 step_executor.go:321] StepExecutor worker(-2): worker exiting due to cancellation
I1012 21:26:56.515015     642 step_executor.go:321] StepExecutor worker(-1): StepExecutor.waitForCompletion(): worker threads all exited
I1012 21:26:56.515348     642 plan_executor.go:254] planExecutor.Execute(...): step executor has completed
I1012 21:26:56.515704     642 plan_executor.go:131] planExecutor.Execute(...): exiting provider canceller
I1012 21:26:56.518059     642 eventsink.go:59] RegisterResource RPC prepared: t=kubernetes:core/v1:Namespace, name=jetpack
I1012 21:26:56.529640     642 provider_plugin.go:472] Provider[aws, 0xc00064f8c0].Configure() failed: err=transport is closing
I1012 21:26:56.538853     642 langruntime_plugin.go:164] langhost[nodejs].Run(pwd=/scratch/provisioning/services,program=.,...,dryrun=true) failed: err=transport is closing
I1012 21:26:56.539006     642 ignore.go:44] Explicitly ignoring and discarding error: rpc error: code = Canceled desc = grpc: the client connection is closing
I1012 21:26:56.553364     642 source_eval.go:178] EvalSourceIterator ended with an error: transport is closing
I1012 21:26:56.553440     642 plan_executor.go:200] planExecutor.Execute(...): incoming events goroutine exiting
I think the culprit is the following: • My pulumi project uses a StackReference to fetch a kubeconfig from a different pulumi project. This kubeconfig tells the current pulumi project where to deploy its resources. • The kubeconfig recently changed. It still points to the same cluster, but it the context name was changed. • I think this is causing pulumi to create a new k8s Provider for these resources, and perhaps it’s trying to create resources in the cluster, but the resources already exist, so it’s failing. I would expect it to fail with a more informative error, though… 😕
^hypothesis confirmed. 
pulumi up
works again after reverting the 
kubeconfig
changes in the upstream project
Open in Slack
PreviousNext
Powered by