https://pulumi.com logo
Title
q

quick-garden-9190

03/07/2023, 10:34 PM
Hello team 😛ulumi-logo: ! Around 4-5 hours ago (18:00 UTC) we started to see consistent 403 response codes when attempting to download certain plugins via the CLI. (checkly) We're observing this in both CI and local environments, CI being amd64, local being arm64. Possible something is going on with the registry? Or potentially something with our pulumi login? Thanks!
s

steep-sunset-89396

03/08/2023, 5:27 AM
@quick-garden-9190 @billions-glass-17089 what happens when you run
pulumi plugin install resource checkly --verbose=9 --logtostderr
?
q

quick-garden-9190

03/08/2023, 5:33 AM
Will try this in a moment. Was able to mitigate the issue by including
--server="<https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2>"
This is the response if I don't specify a version:
I0307 22:34:55.737564   54426 log.go:71] plugin GitHub releases url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>
I0307 22:34:55.737663   54426 log.go:71] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>
I0307 22:34:55.947766   54426 log.go:71] defaultSink::Error(error: 401 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>)
error: 401 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-checkly/releases/latest>
Here's the response if I specify version
1.1.2
I0307 22:35:45.096780   54442 log.go:71] SelectCompatiblePlugin(..., checkly): beginning
I0307 22:35:45.096843   54442 log.go:71] SelectCompatiblePlugin(..., checkly): failed to find match
I0307 22:35:45.096852   54442 log.go:71] defaultSink::Infoerr([resource plugin checkly-1.1.2] installing)
[resource plugin checkly-1.1.2] installing
I0307 22:35:45.096953   54442 log.go:71] plugin GitHub releases url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/tags/v1.1.2>
I0307 22:35:45.096966   54442 log.go:71] full plugin download url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/tags/v1.1.2>
I0307 22:35:45.313735   54442 log.go:71] checkly downloading from <https://get.pulumi.com/releases/plugins>
I0307 22:35:45.313751   54442 log.go:71] checkly downloading from <https://get.pulumi.com/releases/plugins>
I0307 22:35:45.313761   54442 log.go:71] full plugin download url: <https://get.pulumi.com/releases/plugins/pulumi-resource-checkly-v1.1.2-darwin-arm64.tar.gz>
I0307 22:35:45.698466   54442 log.go:71] defaultSink::Error(error: [resource plugin checkly-1.1.2] downloading from : failed to download plugin: checkly-1.1.2: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-checkly-v1.1.2-darwin-arm64.tar.gz>)
error: [resource plugin checkly-1.1.2] downloading from : failed to download plugin: checkly-1.1.2: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-checkly-v1.1.2-darwin-arm64.tar.gz>
s

steep-sunset-89396

03/08/2023, 6:00 AM
Yes, the
checkly
provider is not owned by Pulumi so using
--server
is the way to go for that specific provider. The form you used needs to use a specific version and doesn't understand
latest
. Alternatively,
pulumi plugin install resource checkly --server="<github://api.github.com/checkly>"
q

quick-garden-9190

03/08/2023, 6:06 AM
Noted. Up until earlier today we were happy to let Pulumi automatically handle plugin downloads for us. Any idea why that stopped working?
s

steep-sunset-89396

03/08/2023, 6:08 AM
No I don't know what happening there. I'm not the most familiar with this to be honest. I'm raising an issue to at least improve our docs, but also error handling and clearer error messages.
q

quick-garden-9190

03/08/2023, 6:17 AM
I see. We've been able to mitigate this issue by adding a custom
pulumi plugin install --server=*
step in CI before pulumi update/preview We download plugins on every CI run (😅) and noticed the first failure with checkly plugin around 18:00 UTC.
Noticed the same 403 response code associated with another plugin a few hours later
s

steep-sunset-89396

03/08/2023, 6:20 AM
Is this still happening for you? Do you have a specific command line for that?
oddly enough, I ran
npm install @checkly/pulumi
And both the package and plugin were downloaded correctly
q

quick-garden-9190

03/08/2023, 6:21 AM
pulumi plugin install resource checkly 1.1.2
pulumi plugin install resource onepassword 1.0.5
Both failing for me at the moment.
Npm install works fine for me as well
However, even with the package installed, pulumi will still attempt to download the plugin and fail
s

steep-sunset-89396

03/08/2023, 6:22 AM
How come you have to run the
pulumi plugin install
command by the way ?
q

quick-garden-9190

03/08/2023, 6:22 AM
Unless I mitigate the issue by calling our specific pulumi plugin install command
s

steep-sunset-89396

03/08/2023, 6:23 AM
Is the
.lock
file present as well ?
And the files in the plugin folder have the correct permissions?
q

quick-garden-9190

03/08/2023, 6:23 AM
The initial error message led me to it:
error: could not load plugin for onepassword provider 'urn:pulumi:stage::infrastructure::pulumi:providers:onepassword::onepasswordConnect': Could not automatically download and install resource plugin 'pulumi-resource-onepassword' at version v1.0.5, install the plugin using `pulumi plugin install resource onepassword v1.0.5`.
Underlying error: error downloading plugin onepassword to file: failed to download plugin: onepassword-1.0.5: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-onepassword-v1.0.5-linux-amd64.tar.gz>

Exited with code exit status 255
s

steep-sunset-89396

03/08/2023, 6:23 AM
/home/aureq/.pulumi/plugins/resource-checkly-v1.1.2:
total 39164
-rw-r--r-- 1 aureq users     1343 Mar  8 17:18 CHANGELOG.md
-rw-r--r-- 1 aureq users     1068 Mar  8 17:18 LICENSE
-rwxr-xr-x 1 aureq users 40085187 Mar  8 17:18 pulumi-resource-checkly*
-rw-r--r-- 1 aureq users     5363 Mar  8 17:18 README.md
q

quick-garden-9190

03/08/2023, 6:27 AM
The lockfile is present and is working as configured for plenty of other dependencies
s

steep-sunset-89396

03/08/2023, 6:28 AM
Do you mind opening an issue (and reference that thread) please? I think it's worth for our engineering team to look into this.
q

quick-garden-9190

03/08/2023, 6:29 AM
To be honest I had no idea Pulumi was automatically downloading plugins until today 😂
I assumed they were included with our npm deps
s

steep-sunset-89396

03/08/2023, 6:29 AM
the npm deps automatically download them under the hood as part of a post install hook.
But how they are retrieved is unknown to me
q

quick-garden-9190

03/08/2023, 6:31 AM
Ah I see. I think I came across that mechanism earlier when I was troubleshooting with someone at checkly.
Which is how I found the
--server
workaround
Where can I go to open an issue? assuming here?
submitted
e

enough-air-9670

03/08/2023, 7:28 AM
I have had the same issues today with the AWS provider. After pinning the Pulumi CLI version in the GitHub Action to
3.55.0
(instead of the latest
3.56.0
) it worked first run.
l

limited-rainbow-51650

03/08/2023, 7:33 AM
@quick-garden-9190 the complete commands, given the mentioned providers are not hosted by Pulumi, are these:
pulumi plugin install resource checkly 1.1.2 --server <github://api.github.com/checkly>
pulumi plugin install resource onepassword 1.0.5 --server <github://api.github.com/SimCubeLtd>
This leverages our support for Github Releases. In your case you are most likely hit by Github API rate limiting. If you provide a
GIHTUB_TOKEN
as environment variable, we will send this along for any plugin downloads. Given these are authenticated requests now, your rate limits are higher.
b

billions-glass-17089

03/08/2023, 12:05 PM
The
GITHUB_TOKEN
should be assigned a Personal access token or something else?
q

quick-garden-9190

03/08/2023, 1:50 PM
@limited-rainbow-51650, thanks for the correct command. I'm wondering why I even need the command in the first place since the last 8 months these plugins have automatically downloaded without issue. Like I said earlier, I didn't even know the command was a thing until yesterday.
@worried-balloon-88673, possible you are seeing the same issue as Chris above? I don't use github actions so I didn't experience this symptom, but it smells like the same problem.
w

worried-balloon-88673

03/08/2023, 2:03 PM
i'm not sure if what i'm seeing is the same. i'm getting 401s:
error: could not load plugin for eks provider 'urn:pulumi:barberry::cinder_app::pulumi:providers:eks::default': Could not automatically download and install resource plugin 'pulumi-resource-eks', install the plugin using `pulumi plugin install resource eks`.
  Underlying error: 401 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-eks/releases/latest>
but these occur quite often, it's not a new thing.
q

quick-garden-9190

03/08/2023, 2:03 PM
The consistency of failure is new as of yesterday
I would agree, we see intermittent failures frequently though
l

limited-rainbow-51650

03/08/2023, 2:04 PM
@quick-garden-9190 you are right that in most cases you don’t need to run these commands manually.
q

quick-garden-9190

03/08/2023, 2:09 PM
@worried-balloon-88673, you get 401's if you don't specify a version? What if you use the pulumi command and specify a version, do you get a 403 then? Example:
pulumi plugin install resource checkly 1.1.2
Fails for me with 403
If I exclude the version I see a 401 returned
w

worried-balloon-88673

03/08/2023, 2:11 PM
yep, 401 when i don't specify a version (which seems to be what's happening implicitly when we run the
pulumi/actions@v4
github action).
q

quick-garden-9190

03/08/2023, 2:11 PM
And 403 when you do? Just want to make sure I'm not going insane 😂
w

worried-balloon-88673

03/08/2023, 2:12 PM
it seems to work fine when i specify a version locally!
q

quick-garden-9190

03/08/2023, 2:12 PM
And this started happening yesterday?
w

worried-balloon-88673

03/08/2023, 2:13 PM
i noticed it today, a few hours ago -- but i hadn't tried to deploy anything for the 20 hours before that
q

quick-garden-9190

03/08/2023, 2:21 PM
@limited-rainbow-51650, you suggested I'm being rate limited by Github. But in the verbose logging suggested by @steep-sunset-89396 I don't see any communication with github servers. Are you seeing anything on your side?
l

limited-rainbow-51650

03/08/2023, 2:31 PM
@quick-garden-9190 this is the confusing part currently. If you are being rate limited, we get an HTTP error (now I can’t recall whether that is a 401 or 403). Pulumi then continues with a fallback to scenario trying
<http://get.pulumi.com|get.pulumi.com>
, where thirdparty providers aren’t listed. The user get’s to see the error on the last server. From the snippet higher up, I notice this line:
I0307 22:35:45.096953   54442 log.go:71] plugin GitHub releases url: <https://api.github.com/repos/pulumi/pulumi-checkly/releases/tags/v1.1.2>
with the Github organization being
pulumi
and not
checkly
. We have a bug report where the context doesn’t have the download URL anymore, but it hasn’t been resolved yet. Besides all of this, I see messages of people using our Pulumi Github Actions v4 and encountering this download error problem. More investigation will be needed.
q

quick-garden-9190

03/08/2023, 2:33 PM
Great! We have a mitigation in place, manually calling
pulumi plugin install --server
before executing preview/update. Just wondering why all of a sudden the Pulumi registry isn't returning what we would expect. Thanks for the investigation!
s

straight-fireman-55591

03/08/2023, 4:58 PM
Seems I'm having the same issue. Asked on getting-started channel.
b

billions-glass-17089

03/08/2023, 9:10 PM
pulumi plugin install --server twingate
for us is not fixing the problem
q

quick-garden-9190

03/08/2023, 9:10 PM
Here's complete example command:
pulumi plugin install --server="<https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2>" resource checkly 1.1.2
b

billions-glass-17089

03/08/2023, 9:11 PM
ah
great that did it thank you!