https://pulumi.com
Join Slack
This message was deleted.
# general
s
This message was deleted.
m
pulumi refresh
warns the following:
Copy code
warning: Refreshed resource is in an unhealthy state:
    * Resource 'compiler-deployment' was created but failed to initialize
    * [ProgressDeadlineExceeded] ReplicaSet "compiler-deployment-7f9fbbdf4f" has timed out progressing.
    * Minimum number of Pods to consider the application live was not attained
oh well… i just destroyed the stack and re-created it… problem seems to have gone away now. but im really scared it’ll happen again if i try to make another failed deployment :(
g
Can you share the full log you got from the 
up
? I suspect there's another error message in the output that will give us more information.
m
So… this problem came back again today 😢 let me see if I can share the full log from 
pulumi up
Some interesting output, but not sure if root cause:
Copy code
I1012 21:26:46.600447     642 plugins.go:100] gatherPluginsFromSnapshot(): gathering plugins from snapshot
I1012 21:26:46.600524     642 plugins.go:117] gatherPluginsFromSnapshot(): plugin aws 3.2.1 is required by first-class provider "urn:pulumi:dev::services::pulumi:providers:aws::default_3_2_1"
I1012 21:26:46.600994     642 plugins.go:117] gatherPluginsFromSnapshot(): plugin pulumi <nil> is required by first-class provider "urn:pulumi:dev::services::pulumi:providers:pulumi::default"
I1012 21:26:46.601131     642 plugins.go:117] gatherPluginsFromSnapshot(): plugin kubernetes <nil> is required by first-class provider "urn:pulumi:dev::services::pulumi:providers:kubernetes::infra-cluster-provider"
I1012 21:26:46.601407     642 plugins.go:132] ensurePluginsAreInstalled(): beginning
I1012 21:26:46.601550     642 plugins.go:461] GetPluginPath(language, nodejs, <nil>): found on $PATH /usr/local/bin/pulumi-language-nodejs
I1012 21:26:46.601641     642 plugins.go:137] ensurePluginsAreInstalled(): plugin nodejs <nil> already installed
I1012 21:26:46.602606     642 plugins.go:500] GetPluginPath(resource, aws, 3.2.1): enabling new plugin behavior
I1012 21:26:46.602666     642 plugins.go:582] SelectCompatiblePlugin(..., aws): beginning
I1012 21:26:46.602696     642 plugins.go:611] SelectCompatiblePlugin(..., aws): best plugin aws-3.2.1: semver match
I1012 21:26:46.602713     642 plugins.go:624] SelectCompatiblePlugin(..., aws): selecting plugin 'aws-3.2.1': best match
I1012 21:26:46.602765     642 plugins.go:547] GetPluginPath(resource, aws, 3.2.1): found in cache at /home/toast/.pulumi/plugins/resource-aws-v3.2.1/pulumi-resource-aws
I1012 21:26:46.602819     642 plugins.go:137] ensurePluginsAreInstalled(): plugin aws 3.2.1 already installed
I1012 21:26:46.604009     642 plugins.go:145] ensurePluginsAreInstalled(): plugin pulumi <nil> not installed, doing install
I1012 21:26:46.604128     642 plugins.go:164] installPlugin(pulumi, <nil>): beginning install
I1012 21:26:46.604446     642 plugins.go:530] GetPluginPath(resource, kubernetes, <nil>): found candidate (#2.6.0)
I1012 21:26:46.604507     642 plugins.go:547] GetPluginPath(resource, kubernetes, <nil>): found in cache at /home/toast/.pulumi/plugins/resource-kubernetes-v2.6.0/pulumi-resource-kubernetes
I1012 21:26:46.604589     642 plugins.go:137] ensurePluginsAreInstalled(): plugin kubernetes <nil> already installed
I1012 21:26:46.785635     642 plugins.go:152] ensurePluginsAreInstalled(): completed
I1012 21:26:46.785731     642 update.go:229] newUpdateSource(): failed to install missing plugins: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-pulumi-v%3Cnil%3E-linux-amd64.tar.gz>
I1012 21:26:46.785818     642 plugins.go:220] computeDefaultProviderPlugins(): language host reported empty set of provider plugins, using all plugins
I1012 21:26:46.786048     642 plugins.go:239] computeDefaultProviderPlugins(): considering nodejs
I1012 21:26:46.786331     642 plugins.go:239] computeDefaultProviderPlugins(): considering pulumi
I1012 21:26:46.786366     642 plugins.go:270] computeDefaultProviderPlugins(): plugin pulumi selected for package pulumi (first seen)
I1012 21:26:46.786614     642 plugins.go:239] computeDefaultProviderPlugins(): considering kubernetes
I1012 21:26:46.786648     642 plugins.go:270] computeDefaultProviderPlugins(): plugin kubernetes selected for package kubernetes (first seen)
I1012 21:26:46.786677     642 plugins.go:239] computeDefaultProviderPlugins(): considering aws-3.2.1
I1012 21:26:46.786774     642 plugins.go:270] computeDefaultProviderPlugins(): plugin aws-3.2.1 selected for package aws (first seen)
I1012 21:26:46.786810     642 plugins.go:276] computeDefaultProviderPlugins(): summary of default plugins:
I1012 21:26:46.786865     642 plugins.go:278]   pulumi          = <nil>
I1012 21:26:46.787057     642 plugins.go:278]   kubernetes      = <nil>
I1012 21:26:46.787729     642 plugins.go:278]   aws             = 3.2.1
It’s weird that it tried to download 
<https://get.pulumi.com/releases/plugins/pulumi-resource-pulumi-v%3Cnil%3E-linux-amd64.tar.gz>
the actual failure:
Copy code
I1012 21:26:56.512666     642 source_eval.go:158] EvalSourceIterator produced a registration: t=kubernetes:core/v1:Namespace,name=jetpack,#props=3
I1012 21:26:56.513188     642 plan_executor.go:219] planExecutor.Execute(...): incoming event (nil? false, <nil>)
I1012 21:26:56.513353     642 plan_executor.go:364] planExecutor.handleSingleEvent(...): received RegisterResourceEvent
I1012 21:26:56.513478     642 registry.go:146] GetProvider(urn:pulumi:dev::services::pulumi:providers:kubernetes::infra-cluster-provider::624d6186-2b3e-4ef3-ad02-7847fb4f14d6)
I1012 21:26:56.513754     642 provider_plugin.go:488] Provider[kubernetes, 0xc000b2b4a0].Check(urn:pulumi:dev::services::kubernetes:core/v1:Namespace::jetpack) executing (#olds=3,#news=3
I1012 21:26:56.514106     642 plan_executor.go:237] planExecutor.Execute(...): error handling event: transport is closing
I1012 21:26:56.514342     642 eventsink.go:86] eventSink::Error(<{%reset%}>transport is closing<{%reset%}>)
I1012 21:26:56.514557     642 step_executor.go:321] StepExecutor worker(-1): StepExecutor.waitForCompletion(): waiting for worker threads to exit
I1012 21:26:56.514808     642 step_executor.go:321] StepExecutor worker(-2): worker exiting due to cancellation
I1012 21:26:56.515015     642 step_executor.go:321] StepExecutor worker(-1): StepExecutor.waitForCompletion(): worker threads all exited
I1012 21:26:56.515348     642 plan_executor.go:254] planExecutor.Execute(...): step executor has completed
I1012 21:26:56.515704     642 plan_executor.go:131] planExecutor.Execute(...): exiting provider canceller
I1012 21:26:56.518059     642 eventsink.go:59] RegisterResource RPC prepared: t=kubernetes:core/v1:Namespace, name=jetpack
I1012 21:26:56.529640     642 provider_plugin.go:472] Provider[aws, 0xc00064f8c0].Configure() failed: err=transport is closing
I1012 21:26:56.538853     642 langruntime_plugin.go:164] langhost[nodejs].Run(pwd=/scratch/provisioning/services,program=.,...,dryrun=true) failed: err=transport is closing
I1012 21:26:56.539006     642 ignore.go:44] Explicitly ignoring and discarding error: rpc error: code = Canceled desc = grpc: the client connection is closing
I1012 21:26:56.553364     642 source_eval.go:178] EvalSourceIterator ended with an error: transport is closing
I1012 21:26:56.553440     642 plan_executor.go:200] planExecutor.Execute(...): incoming events goroutine exiting
I think the culprit is the following: • My pulumi project uses a StackReference to fetch a kubeconfig from a different pulumi project. This kubeconfig tells the current pulumi project where to deploy its resources. • The kubeconfig recently changed. It still points to the same cluster, but it the context name was changed. • I think this is causing pulumi to create a new k8s Provider for these resources, and perhaps it’s trying to create resources in the cluster, but the resources already exist, so it’s failing. I would expect it to fail with a more informative error, though… 😕
^hypothesis confirmed. 
pulumi up
works again after reverting the 
kubeconfig
changes in the upstream project
Open in Slack
PreviousNext
Powered by

Pulumi Community

No matter how you like to participate in developer communities, Pulumi wants to meet you there. If you want to meet other Pulumi users to share use-cases and best practices, contribute code or documentation, see us at an event, or just tell a story about something cool you did with Pulumi, you are part of our community.

Powered by