https://pulumi.com logo
Join Slack
Powered by
Hello team :pulumi-logo: ! Around 4-5 hours ago (...
# general
q
Hello team pulumi logo ! Around 4-5 hours ago (18:00 UTC) we started to see consistent 403 response codes when attempting to download certain plugins via the CLI. (checkly) We're observing this in both CI and local environments, CI being amd64, local being arm64. Possible something is going on with the registry? Or potentially something with our pulumi login? Thanks!
s
@quick-garden-9190 @billions-glass-17089 what happens when you run 
pulumi plugin install resource checkly --verbose=9 --logtostderr
?
q
Will try this in a moment. Was able to mitigate the issue by including 
--server="<https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2>"
This is the response if I don't specify a version:
Copy code
I0307 22:34:55.737564   54426 log.go:71] plugin GitHub releases url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>
I0307 22:34:55.737663   54426 log.go:71] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>
I0307 22:34:55.947766   54426 log.go:71] defaultSink::Error(error: 401 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>)
error: 401 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>
Here's the response if I specify version 
1.1.2
Copy code
I0307 22:35:45.096780   54442 log.go:71] SelectCompatiblePlugin(..., checkly): beginning
I0307 22:35:45.096843   54442 log.go:71] SelectCompatiblePlugin(..., checkly): failed to find match
I0307 22:35:45.096852   54442 log.go:71] defaultSink::Infoerr([resource plugin checkly-1.1.2] installing)
[resource plugin checkly-1.1.2] installing
I0307 22:35:45.096953   54442 log.go:71] plugin GitHub releases url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/tags/v1.1.2>
I0307 22:35:45.096966   54442 log.go:71] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/tags/v1.1.2>
I0307 22:35:45.313735   54442 log.go:71] checkly downloading from <https://get.pulumi.com/releases/plugins>
I0307 22:35:45.313751   54442 log.go:71] checkly downloading from <https://get.pulumi.com/releases/plugins>
I0307 22:35:45.313761   54442 log.go:71] full plugin download url: <https://get.pulumi.com/releases/plugins/pulumi-resource-checkly-v1.1.2-darwin-arm64.tar.gz>
I0307 22:35:45.698466   54442 log.go:71] defaultSink::Error(error: [resource plugin checkly-1.1.2] downloading from : failed to download plugin: checkly-1.1.2: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-checkly-v1.1.2-darwin-arm64.tar.gz>)
error: [resource plugin checkly-1.1.2] downloading from : failed to download plugin: checkly-1.1.2: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-checkly-v1.1.2-darwin-arm64.tar.gz>
s
Yes, the 
checkly
provider is not owned by Pulumi so using 
--server
is the way to go for that specific provider. The form you used needs to use a specific version and doesn't understand 
latest
. Alternatively,
Copy code
pulumi plugin install resource checkly --server="<github://api.github.com/checkly>"
q
Noted. Up until earlier today we were happy to let Pulumi automatically handle plugin downloads for us. Any idea why that stopped working?
s
No I don't know what happening there. I'm not the most familiar with this to be honest. I'm raising an issue to at least improve our docs, but also error handling and clearer error messages.
q
I see. We've been able to mitigate this issue by adding a custom 
pulumi plugin install --server=*
step in CI before pulumi update/preview We download plugins on every CI run (😅) and noticed the first failure with checkly plugin around 18:00 UTC.
Noticed the same 403 response code associated with another plugin a few hours later
s
Is this still happening for you? Do you have a specific command line for that?
oddly enough, I ran
Copy code
npm install @checkly/pulumi
And both the package and plugin were downloaded correctly
q
Copy code
pulumi plugin install resource checkly 1.1.2
Copy code
pulumi plugin install resource onepassword 1.0.5
Both failing for me at the moment.
Npm install works fine for me as well
However, even with the package installed, pulumi will still attempt to download the plugin and fail
s
How come you have to run the 
pulumi plugin install
command by the way ?
q
Unless I mitigate the issue by calling our specific pulumi plugin install command
s
Is the 
.lock
file present as well ?
And the files in the plugin folder have the correct permissions?
q
The initial error message led me to it:
Copy code
error: could not load plugin for onepassword provider 'urn:pulumi:stage::infrastructure::pulumi:providers:onepassword::onepasswordConnect': Could not automatically download and install resource plugin 'pulumi-resource-onepassword' at version v1.0.5, install the plugin using `pulumi plugin install resource onepassword v1.0.5`.
Underlying error: error downloading plugin onepassword to file: failed to download plugin: onepassword-1.0.5: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-onepassword-v1.0.5-linux-amd64.tar.gz>

Exited with code exit status 255
s
Copy code
/home/aureq/.pulumi/plugins/resource-checkly-v1.1.2:
total 39164
-rw-r--r-- 1 aureq users     1343 Mar  8 17:18 CHANGELOG.md
-rw-r--r-- 1 aureq users     1068 Mar  8 17:18 LICENSE
-rwxr-xr-x 1 aureq users 40085187 Mar  8 17:18 pulumi-resource-checkly*
-rw-r--r-- 1 aureq users     5363 Mar  8 17:18 README.md
q
The lockfile is present and is working as configured for plenty of other dependencies
s
Do you mind opening an issue (and reference that thread) please? I think it's worth for our engineering team to look into this.
q
To be honest I had no idea Pulumi was automatically downloading plugins until today 😂
I assumed they were included with our npm deps
s
the npm deps automatically download them under the hood as part of a post install hook.
But how they are retrieved is unknown to me
q
Ah I see. I think I came across that mechanism earlier when I was troubleshooting with someone at checkly.
Which is how I found the 
--server
workaround
Where can I go to open an issue? assuming here?
submitted
e
I have had the same issues today with the AWS provider. After pinning the Pulumi CLI version in the GitHub Action to 
3.55.0
(instead of the latest 
3.56.0
) it worked first run.
l
@quick-garden-9190 the complete commands, given the mentioned providers are not hosted by Pulumi, are these:
Copy code
pulumi plugin install resource checkly 1.1.2 --server <github://api.github.com/checkly>
pulumi plugin install resource onepassword 1.0.5 --server <github://api.github.com/SimCubeLtd>
This leverages our support for Github Releases. In your case you are most likely hit by Github API rate limiting. If you provide a 
GIHTUB_TOKEN
as environment variable, we will send this along for any plugin downloads. Given these are authenticated requests now, your rate limits are higher.
b
The 
GITHUB_TOKEN
should be assigned a Personal access token or something else?
q
@limited-rainbow-51650, thanks for the correct command. I'm wondering why I even need the command in the first place since the last 8 months these plugins have automatically downloaded without issue. Like I said earlier, I didn't even know the command was a thing until yesterday.
@worried-balloon-88673, possible you are seeing the same issue as Chris above? I don't use github actions so I didn't experience this symptom, but it smells like the same problem.
w
i'm not sure if what i'm seeing is the same. i'm getting 401s:
Copy code
error: could not load plugin for eks provider 'urn:pulumi:barberry::cinder_app::pulumi:providers:eks::default': Could not automatically download and install resource plugin 'pulumi-resource-eks', install the plugin using `pulumi plugin install resource eks`.
  Underlying error: 401 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-eks/releases/latest>
but these occur quite often, it's not a new thing.
q
The consistency of failure is new as of yesterday
I would agree, we see intermittent failures frequently though
l
@quick-garden-9190 you are right that in most cases you don’t need to run these commands manually.
q
@worried-balloon-88673, you get 401's if you don't specify a version? What if you use the pulumi command and specify a version, do you get a 403 then? Example: 
pulumi plugin install resource checkly 1.1.2
Fails for me with 403
If I exclude the version I see a 401 returned
w
yep, 401 when i don't specify a version (which seems to be what's happening implicitly when we run the 
pulumi/actions@v4
github action).
q
And 403 when you do? Just want to make sure I'm not going insane 😂
w
it seems to work fine when i specify a version locally!
q
And this started happening yesterday?
w
i noticed it today, a few hours ago -- but i hadn't tried to deploy anything for the 20 hours before that
q
@limited-rainbow-51650, you suggested I'm being rate limited by Github. But in the verbose logging suggested by @steep-sunset-89396 I don't see any communication with github servers. Are you seeing anything on your side?
l
@quick-garden-9190 this is the confusing part currently. If you are being rate limited, we get an HTTP error (now I can’t recall whether that is a 401 or 403). Pulumi then continues with a fallback to scenario trying 
<http://get.pulumi.com|get.pulumi.com>
, where thirdparty providers aren’t listed. The user get’s to see the error on the last server. From the snippet higher up, I notice this line:
Copy code
I0307 22:35:45.096953   54442 log.go:71] plugin GitHub releases url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/tags/v1.1.2>
with the Github organization being 
pulumi
and not 
checkly
. We have a bug report where the context doesn’t have the download URL anymore, but it hasn’t been resolved yet. Besides all of this, I see messages of people using our Pulumi Github Actions v4 and encountering this download error problem. More investigation will be needed.
q
Great! We have a mitigation in place, manually calling 
pulumi plugin install --server
before executing preview/update. Just wondering why all of a sudden the Pulumi registry isn't returning what we would expect. Thanks for the investigation!
s
Seems I'm having the same issue. Asked on getting-started channel.
b
pulumi plugin install --server twingate
for us is not fixing the problem
q
Here's complete example command:
Copy code
pulumi plugin install --server="<https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2>" resource checkly 1.1.2
b
ah
great that did it thank you!
Open in Slack
PreviousNext
Powered by