https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# kubernetes
s
This message was deleted.
d
The Release was pulled for a bug. Updating to 4.5.1+ will resolve the issue
f
ahh Jesus… is there some anouncement somewhe? it broke some of my stacks and had to redeploy
d
There wasn't
f
now i guess it will be the same
Copy code
- cluster  : "[recacted]--k8s-europe-west4-dev"
      - context  : "[recacted]-k8s-europe-west4-dev"
      - namespace: "default"
      ~ version  : "4.5.0" => "4.5.3"
removes those
v4.5.0 added them
d
Yes, those outputs caused problems for already existing resources. See here from the 4.5.1 release: https://github.com/pulumi/pulumi-kubernetes/issues/2626
f
ahh dammit… so updating now wont cause replace and delete afterwards?
d
I'm unsure, sorry. I've not seen any reports from others that updating does cause replacements though
f
last week v4.5.0 cased replace on almost all resources which finished quite quickly, but initiated delete afterwards because nothing was replaced actally and deleted the “old” stuff
d
So it didn't replace the resources, just deleted them?
f
yap, in stage 2… went through all of them and marked them as “replaced”… then stage 2 actually deleted them…. had to refresh afterwards and initiate another 
pulumi up
luckilly it was only one project but the 
prod
one took like 2h to complete///
d
Sorry that you got caught out by it. They're going to be doing a post-mortem to improve testing around provider updates for future releases. For updating to 4.5.3, it's worth backing up your stack (
pulumi stack export
, 
... import
) , then doing a targeted update on the provider resource, with a refresh afterwards.
f
development
seems fine with `v4.5.3`… already have stack state backup job, thanks for the heads up!
d
That's good to hear
k
Yeah I got hit with this too right before a demo to the team. Luckily nothing in production.
w
https://pulumi-community.slack.com/archives/CRFURDVQB/p1698835286846269?thread_ts=1698834642.480879&cid=CRFURDVQB @dry-keyboard-94795 I don't understand the motivation behind removing that file from the bucket. It was not well communicated. Our deployment pipeline worked with 4.5.0 and then it stopped working. We had rough time understanding where it's coming from.
d
@wide-dress-96388, The changes in 4.5.0 caused many resources to recreate, so some of the changes were reverted. I guess the guaranteed recreations from this release was enough to have the pulumi team pull the release. https://github.com/pulumi/pulumi-kubernetes/issues/2626 They're discussing it internally, so hopefully there's a better answer on this and future version yanking soon
👍 1
w
This also resulted in weird state now. Because we have previously deployed with 4.5.0 so now all deployments are blocked because some parts of the Pulumi state includes references to 4.5.0 and it cannot be downloaded
Copy code
error: could not load provider for resource urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen|helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen>: could not create provider urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7: load plugin for kubernetes provider 'urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0': Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-darwin-arm64.tar.gz>
d
This is after upgrading to 4.5.1+ and trying to update?
w
Yeah I updated to 4.5.3, and running 4.5.3 right now. Still getting that error on missing 4.5.0
I guess because some resources were created in 4.5.0, so Pulumi needs that as well during 
up
?
I can see things like this in the state JSON file:
Copy code
{
                "urn": "urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0",
                "custom": true,
                "id": "b455c422-8471-431d-990b-76612e505ab7",
                "type": "pulumi:providers:kubernetes",
                "inputs": {
                    "cluster": "guild-cluster",
                    "context": "guild-cluster",
                    "enableServerSideApply": "false",
                    "namespace": "default",
                    "version": "4.5.0"
                },
                "outputs": {
                    "cluster": "guild-cluster",
                    "context": "guild-cluster",
                    "enableServerSideApply": "false",
                    "namespace": "default",
                    "version": "4.5.0"
                },
                "created": "2023-10-24T15:37:11.663647842Z",
                "modified": "2023-10-24T15:37:11.663647842Z"
            },
Copy code
"parent": "urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart::contour-proxy|helm.sh/v3:Chart::contour-proxy>",
                "provider": "urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7",
                "propertyDependencies": {
                    "apiVersion": null,
                    "kind": null,
                    "metadata": null,
                    "spec": null
                },
                "modified": "2023-09-26T14:11:29.771465Z",
                "sourcePosition": "project:///node_modules/.pnpm/@pulumi+kubernetes@4.5.0/node_modules/@pulumi/apiextensions/v1/customResourceDefinition.ts#90,9"
f
remove node_modules and do a clean install with v4.5.3… even clear npm cache if you can, the new one is not registered properly it seems, still seeing the old one
d
Is the stack set to refresh when updating? Try with 
pulumi update --refresh=false
When refreshing, it'll still try to use the old provider in state, whereas update will only look at the new provider
w
remove node_modules and do a clean install with v4.5.3… even clear npm cache if you can, the new one is not registered properly it seems, still seeing the old one
I already tried to remove node_modules. I don't have any trace of 
4.5.0
in my env. This also happens in our CI where we install all deps from scratch.
I'm trying to run now without refresh, thanks @dry-keyboard-94795
same error
Copy code
error: could not load provider for resource urn:pulumi:prod::hive::kubernetes:<http://helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen|helm.sh/v3:Chart$kubernetes:batch/v1:Job::contour/contour-proxy-contour-certgen>: could not create provider urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0::b455c422-8471-431d-990b-76612e505ab7: load plugin for kubernetes provider 'urn:pulumi:prod::hive::pulumi:providers:kubernetes::default_4_5_0': Could not automatically download and install resource plugin 'pulumi-resource-kubernetes' at version v4.5.0, install the plugin using `pulumi plugin install resource kubernetes v4.5.0`: error downloading provider kubernetes to file: failed to download plugin: kubernetes-4.5.0: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v4.5.0-darwin-arm64.tar.gz>
maybe 
preview
also needs the older provider? trying now also with 
--skip-preview
d
Can you provide the outputs of:
Copy code
pulumi about
pulumi plugin ls
pulumi plugin install
w
sure. here you go. the 
pulumi plugin install
has no output
about.txtplugin_ls.txt
btw also tried with 
--skip-preview
and i got the same error as before. Seems like Pulumi still tries to download the old provider during update
Btw... I have another workstation that has 
4.5.0
installed... should I try to deploy from it locally? 🤔
f
yes, i was about to suggest that… at least for 
refresh
my old plugins are never cleaned after update btw, that might help
Copy code
kubernetes     resource  4.5.3    92 MB   1 day ago      1 day ago
kubernetes     resource  4.5.0    92 MB   1 week ago     1 week ago
kubernetes     resource  4.4.0    92 MB   1 week ago     1 week ago
kubernetes     resource  4.3.0    91 MB   1 week ago     1 week ago
kubernetes     resource  4.2.0    91 MB   2 weeks ago    5 days ago
kubernetes     resource  4.1.1    90 MB   2 months ago   2 months ago
kubernetes     resource  4.1.0    90 MB   2 months ago   2 months ago
kubernetes     resource  4.0.3    90 MB   3 months ago   3 months ago
kubernetes     resource  4.0.2    90 MB   3 months ago   3 months ago
kubernetes     resource  4.0.1    90 MB   3 months ago   3 months ago
kubernetes     resource  3.30.2   89 MB   3 months ago   3 months ago
kubernetes     resource  3.30.1   89 MB   4 months ago   2 months ago
kubernetes     resource  3.30.0   89 MB   4 months ago   4 months ago
kubernetes     resource  3.29.0   89 MB   4 months ago   4 months ago
kubernetes     resource  3.28.1   89 MB   5 months ago   5 months ago
kubernetes     resource  3.27.1   88 MB   5 months ago   5 months ago
kubernetes     resource  3.25.0   86 MB   6 months ago   6 months ago
kubernetes     resource  3.24.2   86 MB   7 months ago   7 months ago
kubernetes     resource  3.24.1   86 MB   8 months ago   8 months ago
kubernetes     resource  3.24.0   86 MB   8 months ago   8 months ago
kubernetes     resource  3.23.1   83 MB   9 months ago   9 months ago
kubernetes     resource  3.23.0   83 MB   10 months ago  10 months ago
w
Yeah I'm trying that now. But that's really far from ideal. I think Pulumi should never delete the providers tar files, even if they are having bugs
f
yap, even marking is as 
deprecated
on NPM should fix the issue, instead of just nuking it right away
➕ 1
d
Yeah I'm trying that now. But that's really far from ideal. I think Pulumi should never delete the providers tar files, even if they are having bugs
This has been raised :)
✅ 1
And yes, the simplest thing will be to use the other workstation if it has 4.5.0 at this point
w
I was able to run 
pulumi refresh
on the other workstation (that has 4.5.0). Trying to run 
up
on that workstation now. If I understand correctly, this should update the pulumi state and remove references to 4.5.0?
d
Provided you've updated to 4.5.3, yes
How did that go, @wide-dress-96388?
w
Trying to solve another issue with the deployment, will do a workaround and will update in a few minutes
👍 1
ok i was able to get a ✅ on running from the old workstation. Going to try to run with the new one (without 4.5.0)
Ok seems to be fine now. I'm going to try to run this from our CI, will update soon 🙂 Thanks @dry-keyboard-94795
🙌 1
d
Good to hear. This has all been raised internally, so hopefully the release retraction process is improved next time it happens
🙌 1
Open in Slack
PreviousNext
Powered by