hello guys what happened to the pulumi kubernetes plugin `v4 Pulumi Community #kubernetes

hello guys, what happened to the pulumi kubernetes...

freezing-quill-32178

11/01/2023, 10:30 AM

hello guys, what happened to the pulumi kubernetes plugin

v4.5.0

the CI started failing with

Copy code

error: Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-linux-amd64.tar.gz>

also i cannot find it in https://github.com/pulumi/pulumi-kubernetes/releases any reason for it?

dry-keyboard-94795

11/01/2023, 10:41 AM

The Release was pulled for a bug. Updating to 4.5.1+ will resolve the issue

freezing-quill-32178

11/01/2023, 10:42 AM

ahh Jesus… is there some anouncement somewhe? it broke some of my stacks and had to redeploy

dry-keyboard-94795

11/01/2023, 10:43 AM

There wasn't

freezing-quill-32178

11/01/2023, 10:44 AM

now i guess it will be the same

Copy code

- cluster  : "[recacted]--k8s-europe-west4-dev"
      - context  : "[recacted]-k8s-europe-west4-dev"
      - namespace: "default"
      ~ version  : "4.5.0" => "4.5.3"

freezing-quill-32178

11/01/2023, 10:44 AM

removes those

freezing-quill-32178

11/01/2023, 10:44 AM

v4.5.0 added them

dry-keyboard-94795

11/01/2023, 10:45 AM

Yes, those outputs caused problems for already existing resources. See here from the 4.5.1 release: https://github.com/pulumi/pulumi-kubernetes/issues/2626

freezing-quill-32178

11/01/2023, 10:49 AM

ahh dammit… so updating now wont cause replace and delete afterwards?

dry-keyboard-94795

11/01/2023, 10:50 AM

I'm unsure, sorry. I've not seen any reports from others that updating does cause replacements though

freezing-quill-32178

11/01/2023, 10:52 AM

last week v4.5.0 cased replace on almost all resources which finished quite quickly, but initiated delete afterwards because nothing was replaced actally and deleted the “old” stuff

dry-keyboard-94795

11/01/2023, 10:54 AM

So it didn't replace the resources, just deleted them?

freezing-quill-32178

11/01/2023, 10:57 AM

yap, in stage 2… went through all of them and marked them as “replaced”… then stage 2 actually deleted them…. had to refresh afterwards and initiate another

pulumi up

freezing-quill-32178

11/01/2023, 10:57 AM

luckilly it was only one project but the

prod

one took like 2h to complete///

dry-keyboard-94795

11/01/2023, 11:03 AM

Sorry that you got caught out by it. They're going to be doing a post-mortem to improve testing around provider updates for future releases. For updating to 4.5.3, it's worth backing up your stack (

pulumi stack export

... import

) , then doing a targeted update on the provider resource, with a refresh afterwards.

freezing-quill-32178

11/01/2023, 12:30 PM

development

seems fine with `v4.5.3`… already have stack state backup job, thanks for the heads up!

dry-keyboard-94795

11/01/2023, 12:31 PM

That's good to hear

kind-fireman-33438

11/01/2023, 2:03 PM

Yeah I got hit with this too right before a demo to the team. Luckily nothing in production.

wide-dress-96388

11/02/2023, 10:01 AM

https://pulumi-community.slack.com/archives/CRFURDVQB/p1698835286846269?thread_ts=1698834642.480879&cid=CRFURDVQB @dry-keyboard-94795 I don't understand the motivation behind removing that file from the bucket. It was not well communicated. Our deployment pipeline worked with 4.5.0 and then it stopped working. We had rough time understanding where it's coming from.

dry-keyboard-94795

11/02/2023, 10:07 AM

@wide-dress-96388, The changes in 4.5.0 caused many resources to recreate, so some of the changes were reverted. I guess the guaranteed recreations from this release was enough to have the pulumi team pull the release. https://github.com/pulumi/pulumi-kubernetes/issues/2626 They're discussing it internally, so hopefully there's a better answer on this and future version yanking soon

wide-dress-96388

11/02/2023, 10:22 AM

This also resulted in weird state now. Because we have previously deployed with 4.5.0 so now all deployments are blocked because some parts of the Pulumi state includes references to 4.5.0 and it cannot be downloaded

Copy code

error: could not load provider for resource urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen|helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen>: could not create provider urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7: load plugin for kubernetes provider 'urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0': Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-darwin-arm64.tar.gz>

dry-keyboard-94795

11/02/2023, 10:24 AM

This is after upgrading to 4.5.1+ and trying to update?

wide-dress-96388

11/02/2023, 10:24 AM

Yeah I updated to 4.5.3, and running 4.5.3 right now. Still getting that error on missing 4.5.0

wide-dress-96388

11/02/2023, 10:25 AM

I guess because some resources were created in 4.5.0, so Pulumi needs that as well during

up

wide-dress-96388

11/02/2023, 10:25 AM

I can see things like this in the state JSON file:

Copy code

{
                "urn": "urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0",
                "custom": true,
                "id": "b455c422-8471-431d-990b-76612e505ab7",
                "type": "pulumi:providers:kubernetes",
                "inputs": {
                    "cluster": "guild-cluster",
                    "context": "guild-cluster",
                    "enableServerSideApply": "false",
                    "namespace": "default",
                    "version": "4.5.0"
                },
                "outputs": {
                    "cluster": "guild-cluster",
                    "context": "guild-cluster",
                    "enableServerSideApply": "false",
                    "namespace": "default",
                    "version": "4.5.0"
                },
                "created": "2023-10-24T15:37:11.663647842Z",
                "modified": "2023-10-24T15:37:11.663647842Z"
            },

Copy code

"parent": "urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart::contour-proxy|helm.sh/v3:Chart::contour-proxy>",
                "provider": "urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7",
                "propertyDependencies": {
                    "apiVersion": null,
                    "kind": null,
                    "metadata": null,
                    "spec": null
                },
                "modified": "2023-09-26T14:11:29.771465Z",
                "sourcePosition": "project:///node_modules/.pnpm/@pulumi+kubernetes@4.5.0/node_modules/@pulumi/apiextensions/v1/customResourceDefinition.ts#90,9"

freezing-quill-32178

11/02/2023, 10:27 AM

remove node_modules and do a clean install with v4.5.3… even clear npm cache if you can, the new one is not registered properly it seems, still seeing the old one

dry-keyboard-94795

11/02/2023, 10:28 AM

Is the stack set to refresh when updating? Try with

pulumi update --refresh=false

dry-keyboard-94795

11/02/2023, 10:28 AM

When refreshing, it'll still try to use the old provider in state, whereas update will only look at the new provider

wide-dress-96388

11/02/2023, 10:29 AM

remove node_modules and do a clean install with v4.5.3… even clear npm cache if you can, the new one is not registered properly it seems, still seeing the old one

I already tried to remove node_modules. I don't have any trace of

4.5.0

in my env. This also happens in our CI where we install all deps from scratch.

wide-dress-96388

11/02/2023, 10:30 AM

I'm trying to run now without refresh, thanks @dry-keyboard-94795

wide-dress-96388

11/02/2023, 10:31 AM

same error

Copy code

error: could not load provider for resource urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen|helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen>: could not create provider urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7: load plugin for kubernetes provider 'urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0': Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-darwin-arm64.tar.gz>

wide-dress-96388

11/02/2023, 10:32 AM

maybe

preview

also needs the older provider? trying now also with

--skip-preview

dry-keyboard-94795

11/02/2023, 10:33 AM

Can you provide the outputs of:

Copy code

pulumi about
pulumi plugin ls
pulumi plugin install

wide-dress-96388

11/02/2023, 10:36 AM

sure. here you go. the

pulumi plugin install

has no output

about.txt plugin_ls.txt

wide-dress-96388

11/02/2023, 10:37 AM

btw also tried with

--skip-preview

and i got the same error as before. Seems like Pulumi still tries to download the old provider during update

wide-dress-96388

11/02/2023, 10:39 AM

Btw... I have another workstation that has

4.5.0

installed... should I try to deploy from it locally? 🤔

freezing-quill-32178

11/02/2023, 10:43 AM

yes, i was about to suggest that… at least for

refresh

my old plugins are never cleaned after update btw, that might help

Copy code

kubernetes     resource  4.5.3    92 MB   1 day ago      1 day ago
kubernetes     resource  4.5.0    92 MB   1 week ago     1 week ago
kubernetes     resource  4.4.0    92 MB   1 week ago     1 week ago
kubernetes     resource  4.3.0    91 MB   1 week ago     1 week ago
kubernetes     resource  4.2.0    91 MB   2 weeks ago    5 days ago
kubernetes     resource  4.1.1    90 MB   2 months ago   2 months ago
kubernetes     resource  4.1.0    90 MB   2 months ago   2 months ago
kubernetes     resource  4.0.3    90 MB   3 months ago   3 months ago
kubernetes     resource  4.0.2    90 MB   3 months ago   3 months ago
kubernetes     resource  4.0.1    90 MB   3 months ago   3 months ago
kubernetes     resource  3.30.2   89 MB   3 months ago   3 months ago
kubernetes     resource  3.30.1   89 MB   4 months ago   2 months ago
kubernetes     resource  3.30.0   89 MB   4 months ago   4 months ago
kubernetes     resource  3.29.0   89 MB   4 months ago   4 months ago
kubernetes     resource  3.28.1   89 MB   5 months ago   5 months ago
kubernetes     resource  3.27.1   88 MB   5 months ago   5 months ago
kubernetes     resource  3.25.0   86 MB   6 months ago   6 months ago
kubernetes     resource  3.24.2   86 MB   7 months ago   7 months ago
kubernetes     resource  3.24.1   86 MB   8 months ago   8 months ago
kubernetes     resource  3.24.0   86 MB   8 months ago   8 months ago
kubernetes     resource  3.23.1   83 MB   9 months ago   9 months ago
kubernetes     resource  3.23.0   83 MB   10 months ago  10 months ago

wide-dress-96388

11/02/2023, 10:43 AM

Yeah I'm trying that now. But that's really far from ideal. I think Pulumi should never delete the providers tar files, even if they are having bugs

freezing-quill-32178

11/02/2023, 10:44 AM

yap, even marking is as

deprecated

on NPM should fix the issue, instead of just nuking it right away

dry-keyboard-94795

11/02/2023, 10:46 AM

Yeah I'm trying that now. But that's really far from ideal. I think Pulumi should never delete the providers tar files, even if they are having bugs

This has been raised :)

dry-keyboard-94795

11/02/2023, 10:47 AM

And yes, the simplest thing will be to use the other workstation if it has 4.5.0 at this point

wide-dress-96388

11/02/2023, 10:52 AM

I was able to run

pulumi refresh

on the other workstation (that has 4.5.0). Trying to run

up

on that workstation now. If I understand correctly, this should update the pulumi state and remove references to 4.5.0?

dry-keyboard-94795

11/02/2023, 10:57 AM

Provided you've updated to 4.5.3, yes

dry-keyboard-94795

11/02/2023, 11:07 AM

How did that go, @wide-dress-96388?

wide-dress-96388

11/02/2023, 11:08 AM

Trying to solve another issue with the deployment, will do a workaround and will update in a few minutes

wide-dress-96388

11/02/2023, 11:14 AM

ok i was able to get a ✅ on running from the old workstation. Going to try to run with the new one (without 4.5.0)

wide-dress-96388

11/02/2023, 11:21 AM

Ok seems to be fine now. I'm going to try to run this from our CI, will update soon 🙂 Thanks @dry-keyboard-94795

dry-keyboard-94795

11/02/2023, 11:52 AM

Good to hear. This has all been raised internally, so hopefully the release retraction process is improved next time it happens

Open in Slack

Previous Next