https://pulumi.com logo
Powered by
#kubernetes
Title
# kubernetes
f

freezing-quill-32178

11/01/2023, 10:30 AM
hello guys, what happened to the pulumi kubernetes plugin 
v4.5.0
the CI started failing with
Copy code
error: Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-linux-amd64.tar.gz>
also i cannot find it in https://github.com/pulumi/pulumi-kubernetes/releases any reason for it?
d

dry-keyboard-94795

11/01/2023, 10:41 AM
The Release was pulled for a bug. Updating to 4.5.1+ will resolve the issue
f

freezing-quill-32178

11/01/2023, 10:42 AM
ahh Jesus… is there some anouncement somewhe? it broke some of my stacks and had to redeploy
d

dry-keyboard-94795

11/01/2023, 10:43 AM
There wasn't
f

freezing-quill-32178

11/01/2023, 10:44 AM
now i guess it will be the same
Copy code
- cluster  : "[recacted]--k8s-europe-west4-dev"
      - context  : "[recacted]-k8s-europe-west4-dev"
      - namespace: "default"
      ~ version  : "4.5.0" => "4.5.3"
removes those
v4.5.0 added them
d

dry-keyboard-94795

11/01/2023, 10:45 AM
Yes, those outputs caused problems for already existing resources. See here from the 4.5.1 release: https://github.com/pulumi/pulumi-kubernetes/issues/2626
f

freezing-quill-32178

11/01/2023, 10:49 AM
ahh dammit… so updating now wont cause replace and delete afterwards?
d

dry-keyboard-94795

11/01/2023, 10:50 AM
I'm unsure, sorry. I've not seen any reports from others that updating does cause replacements though
f

freezing-quill-32178

11/01/2023, 10:52 AM
last week v4.5.0 cased replace on almost all resources which finished quite quickly, but initiated delete afterwards because nothing was replaced actally and deleted the “old” stuff
d

dry-keyboard-94795

11/01/2023, 10:54 AM
So it didn't replace the resources, just deleted them?
f

freezing-quill-32178

11/01/2023, 10:57 AM
yap, in stage 2… went through all of them and marked them as “replaced”… then stage 2 actually deleted them…. had to refresh afterwards and initiate another 
pulumi up
luckilly it was only one project but the 
prod
one took like 2h to complete///
d

dry-keyboard-94795

11/01/2023, 11:03 AM
Sorry that you got caught out by it. They're going to be doing a post-mortem to improve testing around provider updates for future releases. For updating to 4.5.3, it's worth backing up your stack (
pulumi stack export
, 
... import
) , then doing a targeted update on the provider resource, with a refresh afterwards.
f

freezing-quill-32178

11/01/2023, 12:30 PM
development
seems fine with `v4.5.3`… already have stack state backup job, thanks for the heads up!
d

dry-keyboard-94795

11/01/2023, 12:31 PM
That's good to hear
k

kind-fireman-33438

11/01/2023, 2:03 PM
Yeah I got hit with this too right before a demo to the team. Luckily nothing in production.
w

wide-dress-96388

11/02/2023, 10:01 AM
https://pulumi-community.slack.com/archives/CRFURDVQB/p1698835286846269?thread_ts=1698834642.480879&cid=CRFURDVQB @dry-keyboard-94795 I don't understand the motivation behind removing that file from the bucket. It was not well communicated. Our deployment pipeline worked with 4.5.0 and then it stopped working. We had rough time understanding where it's coming from.
d

dry-keyboard-94795

11/02/2023, 10:07 AM
@wide-dress-96388, The changes in 4.5.0 caused many resources to recreate, so some of the changes were reverted. I guess the guaranteed recreations from this release was enough to have the pulumi team pull the release. https://github.com/pulumi/pulumi-kubernetes/issues/2626 They're discussing it internally, so hopefully there's a better answer on this and future version yanking soon
w

wide-dress-96388

11/02/2023, 10:22 AM
This also resulted in weird state now. Because we have previously deployed with 4.5.0 so now all deployments are blocked because some parts of the Pulumi state includes references to 4.5.0 and it cannot be downloaded
Copy code
error: could not load provider for resource urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen|helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen>: could not create provider urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7: load plugin for kubernetes provider 'urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0': Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-darwin-arm64.tar.gz>
d

dry-keyboard-94795

11/02/2023, 10:24 AM
This is after upgrading to 4.5.1+ and trying to update?
w

wide-dress-96388

11/02/2023, 10:24 AM
Yeah I updated to 4.5.3, and running 4.5.3 right now. Still getting that error on missing 4.5.0
I guess because some resources were created in 4.5.0, so Pulumi needs that as well during 
up
?
I can see things like this in the state JSON file:
Copy code
{
                "urn": "urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0",
                "custom": true,
                "id": "b455c422-8471-431d-990b-76612e505ab7",
                "type": "pulumi:providers:kubernetes",
                "inputs": {
                    "cluster": "guild-cluster",
                    "context": "guild-cluster",
                    "enableServerSideApply": "false",
                    "namespace": "default",
                    "version": "4.5.0"
                },
                "outputs": {
                    "cluster": "guild-cluster",
                    "context": "guild-cluster",
                    "enableServerSideApply": "false",
                    "namespace": "default",
                    "version": "4.5.0"
                },
                "created": "2023-10-24T15:37:11.663647842Z",
                "modified": "2023-10-24T15:37:11.663647842Z"
            },
Copy code
"parent": "urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart::contour-proxy|helm.sh/v3:Chart::contour-proxy>",
                "provider": "urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7",
                "propertyDependencies": {
                    "apiVersion": null,
                    "kind": null,
                    "metadata": null,
                    "spec": null
                },
                "modified": "2023-09-26T14:11:29.771465Z",
                "sourcePosition": "project:///node_modules/.pnpm/@pulumi+kubernetes@4.5.0/node_modules/@pulumi/apiextensions/v1/customResourceDefinition.ts#90,9"
f

freezing-quill-32178

11/02/2023, 10:27 AM
remove node_modules and do a clean install with v4.5.3… even clear npm cache if you can, the new one is not registered properly it seems, still seeing the old one
d

dry-keyboard-94795

11/02/2023, 10:28 AM
Is the stack set to refresh when updating? Try with 
pulumi update --refresh=false
When refreshing, it'll still try to use the old provider in state, whereas update will only look at the new provider
w

wide-dress-96388

11/02/2023, 10:29 AM
remove node_modules and do a clean install with v4.5.3… even clear npm cache if you can, the new one is not registered properly it seems, still seeing the old one
I already tried to remove node_modules. I don't have any trace of 
4.5.0
in my env. This also happens in our CI where we install all deps from scratch.
I'm trying to run now without refresh, thanks @dry-keyboard-94795
same error
Copy code
error: could not load provider for resource urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen|helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen>: could not create provider urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7: load plugin for kubernetes provider 'urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0': Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-darwin-arm64.tar.gz>
maybe 
preview
also needs the older provider? trying now also with 
--skip-preview
d

dry-keyboard-94795

11/02/2023, 10:33 AM
Can you provide the outputs of:
Copy code
pulumi about
pulumi plugin ls
pulumi plugin install
w

wide-dress-96388

11/02/2023, 10:36 AM
sure. here you go. the 
pulumi plugin install
has no output
about.txtplugin_ls.txt
btw also tried with 
--skip-preview
and i got the same error as before. Seems like Pulumi still tries to download the old provider during update
Btw... I have another workstation that has 
4.5.0
installed... should I try to deploy from it locally? 🤔
f

freezing-quill-32178

11/02/2023, 10:43 AM
yes, i was about to suggest that… at least for 
refresh
my old plugins are never cleaned after update btw, that might help
Copy code
kubernetes     resource  4.5.3    92 MB   1 day ago      1 day ago
kubernetes     resource  4.5.0    92 MB   1 week ago     1 week ago
kubernetes     resource  4.4.0    92 MB   1 week ago     1 week ago
kubernetes     resource  4.3.0    91 MB   1 week ago     1 week ago
kubernetes     resource  4.2.0    91 MB   2 weeks ago    5 days ago
kubernetes     resource  4.1.1    90 MB   2 months ago   2 months ago
kubernetes     resource  4.1.0    90 MB   2 months ago   2 months ago
kubernetes     resource  4.0.3    90 MB   3 months ago   3 months ago
kubernetes     resource  4.0.2    90 MB   3 months ago   3 months ago
kubernetes     resource  4.0.1    90 MB   3 months ago   3 months ago
kubernetes     resource  3.30.2   89 MB   3 months ago   3 months ago
kubernetes     resource  3.30.1   89 MB   4 months ago   2 months ago
kubernetes     resource  3.30.0   89 MB   4 months ago   4 months ago
kubernetes     resource  3.29.0   89 MB   4 months ago   4 months ago
kubernetes     resource  3.28.1   89 MB   5 months ago   5 months ago
kubernetes     resource  3.27.1   88 MB   5 months ago   5 months ago
kubernetes     resource  3.25.0   86 MB   6 months ago   6 months ago
kubernetes     resource  3.24.2   86 MB   7 months ago   7 months ago
kubernetes     resource  3.24.1   86 MB   8 months ago   8 months ago
kubernetes     resource  3.24.0   86 MB   8 months ago   8 months ago
kubernetes     resource  3.23.1   83 MB   9 months ago   9 months ago
kubernetes     resource  3.23.0   83 MB   10 months ago  10 months ago
w

wide-dress-96388

11/02/2023, 10:43 AM
Yeah I'm trying that now. But that's really far from ideal. I think Pulumi should never delete the providers tar files, even if they are having bugs
f

freezing-quill-32178

11/02/2023, 10:44 AM
yap, even marking is as 
deprecated
on NPM should fix the issue, instead of just nuking it right away
d

dry-keyboard-94795

11/02/2023, 10:46 AM
Yeah I'm trying that now. But that's really far from ideal. I think Pulumi should never delete the providers tar files, even if they are having bugs
This has been raised :)
And yes, the simplest thing will be to use the other workstation if it has 4.5.0 at this point
w

wide-dress-96388

11/02/2023, 10:52 AM
I was able to run 
pulumi refresh
on the other workstation (that has 4.5.0). Trying to run 
up
on that workstation now. If I understand correctly, this should update the pulumi state and remove references to 4.5.0?
d

dry-keyboard-94795

11/02/2023, 10:57 AM
Provided you've updated to 4.5.3, yes
How did that go, @wide-dress-96388?
w

wide-dress-96388

11/02/2023, 11:08 AM
Trying to solve another issue with the deployment, will do a workaround and will update in a few minutes
ok i was able to get a ✅ on running from the old workstation. Going to try to run with the new one (without 4.5.0)
Ok seems to be fine now. I'm going to try to run this from our CI, will update soon 🙂 Thanks @dry-keyboard-94795
d

dry-keyboard-94795

11/02/2023, 11:52 AM
Good to hear. This has all been raised internally, so hopefully the release retraction process is improved next time it happens
Powered by