https://pulumi.com logo
Join Slack
Powered by
Hi all. I'm running into an issue where pulumi rep...
# aws
r
Hi all. I'm running into an issue where pulumi reports both a rate limit issue and 403 when trying to install the aws resource plugin. I reproduce this both when installing the plugin explicitly with 
pulumi plugin install resource aws
and when running 
pulumi preview
for my stack. Oddly, I encounter this only in CI (codebuild with aws/codebuild/standard:7.0; Linux EC2) -- not from my local machine (Mac M2). Also odd is that the plugin installs just fine in CI configured almost identically for other projects in the same AWS account. I saw [this page](https://www.pulumi.com/docs/support/troubleshooting/#403-error-fetching-plugin) on troubleshooting such errors, but I still reproduce even after destroying the stack and trying to preview from scratch in CI. Also, since the issue occurs outside of the project by issuing 
pulumi plugin install
, seems like those instructions might not be relevant. Any ideas? (logs in thread)
installing directly:
Copy code
[Container] 2024/02/14 00:18:27.278435 Running command pulumi plugin install resource aws
E0214 00:18:27.461775     322 plugins.go:430] GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>, try again in 28m49.53822938s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.
error: rate limit exceeded: 403 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>

[Container] 2024/02/14 00:18:27.519855 Command did not exit successfully pulumi plugin install resource aws exit status 255
via pulumi preview:
Copy code
Initializing Pulumi state
AWS_PROFILE=deployment \
pulumi login <s3://bucket-name>
Logged in to ip-10-186-48-50.us-west-2.compute.internal as root (<s3://bucket-name>)
AWS_PROFILE=deployment \
PULUMI_CONFIG_PASSPHRASE= \
pulumi preview --stack dev --config-file=Pulumi.dev.yaml
Previewing update (dev):

@ previewing update.......
    pulumi:pulumi:Stack worker-dev  go: downloading <http://github.com/pulumi/pulumi/sdk/v3|github.com/pulumi/pulumi/sdk/v3> v3.105.0
    pulumi:pulumi:Stack worker-dev  go: downloading <http://github.com/pulumi/pulumi-aws/sdk/v5|github.com/pulumi/pulumi-aws/sdk/v5> v5.43.0
...
@ previewing update..........................................................................E0214 00:22:44.349567    2415 plugins.go:430] GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>, try again in 20m51.650437891s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.

    pulumi:providers:aws default  error: could not find latest version for provider aws: rate limit exceeded: 403 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>
    pulumi:providers:aws default  1 error
s
I'm not sure there's much we can do here, other than suggest you supply a 
GITHUB_TOKEN
value as suggested to authenticate to GitHub and get around the throttling. Are there concerns/issues with that approach?
r
I can try that. However, the other CI jobs I have in the same AWS account on the same vpc also report throttling, but they ultimately succeed, consistently. In those jobs, the rate limit isn't accompanied with a 403 error, and the pulumi job doesn't exit with 255 status. In all cases, we are using pulumi v3.105.0.
Copy code
pulumi preview --stack stage --config-file=Pulumi.stage.yaml
Previewing update (stage):
E0213 22:14:38.142627   11522 plugins.go:430] GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/tags/v5.42.0>, try again in 32m8.857380047s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.
[resource plugin aws-5.42.0] installing
@ previewing update......

    pulumi:pulumi:Stack acme-metadata-server-acme-metadata-server-prod-prod  go: downloading <http://github.com/pulumi/pulumi/sdk/v3|github.com/pulumi/pulumi/sdk/v3> v3.78.1
    pulumi:pulumi:Stack acme-metadata-server-acme-metadata-server-prod-prod  go: downloading <http://github.com/pulumi/pulumi-aws/sdk/v5|github.com/pulumi/pulumi-aws/sdk/v5> v5.42.0
...
Resources:
    ~ 3 to update
    +-1 to replace
    4 changes. 65 unchanged


[Container] 2024/02/13 22:15:40.013638 Phase complete: BUILD State: SUCCEEDED
I saw this thread and thought maybe there was some fall back in place that somehow wasn't employed for the failing job.
s
That's a possibility. This only seems to affect the AWS provider, is that right?
r
That's all I've seen so far.
Just found that adding an aws version to the pulumi plugin install command allowed this failing job to continue. I still get the rate limit, but the preview continues
s
What version did you add, if I may ask? (This might help determine what issues, if any, exist with the current fallback mechanism.)
r
Reproed with both v5.43.0 and v5.42.0
s
Roger that, thanks for the info.
r
Still not sure why the other CI jobs work without having to explicitly install the plugin before running the preview, but I did try adding the explicit install without the version on those jobs, and the issue reproduces there as well. 🤷
s
Got some information from internal teams: “If they don’t supply a version we have to hit the GitHub API to ask what the latest version is. If they’re rate limited then we can’t do that and we’re kinda stuck. We don’t have a way to query 
<http://get.pulumi.com|get.pulumi.com>
for versions available.” This explains why adding the explicit version works, but as you pointed out it still doesn’t explain why some environments need the explicit install and others don’t. (Although I do seem to recall that Pulumi YAML needs explicit plugin installs; are you using YAML at all?)
r
No, we use go for everything.
Looking back up at the logs, I can see that for the jobs that fail we see:
Copy code
GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>
Whereas for the jobs that succeed we see:
Copy code
GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/tags/v5.42.0>
Curious that for the failing job, it tries to get latest. I have to assume the version comes from go.mod/go.sum.
s
To me, the logs are consistent with the explanation; it’s hitting GitHub to determine what the latest is and getting throttled, or it’s trying a specific version, getting throttled, and then falling back to 
<http://get.pulumi.com|get.pulumi.com>
. And yes, I would expect the version to get pulled from 
go.mod
.
r
I guess what I mean is that I'm surprised that https://api.github.com/repos/pulumi/pulumi-aws/releases/latest is ever used instead of https://api.github.com/repos/pulumi/pulumi-aws/releases/tags/v5.42.0, when the version is available in go.mod.
s
Yes, agreed. It does seem there is something missing that is causing it to hit the 
latest
endpoint when the version is included in 
go.mod
.
r
In any case, thanks for your help. If I find anything, I'll post back. If you think of something, let me know.
s
Will do!
r
FYI, I found that if i configured CI to explicitly install the plugin by version and then do pulumi up, everything would work (as already mentioned). Then, if I remove the explicit plugin install from the CI script, pulumi up and pulumi preview will continue to work: it fetches the plugin by version, falling back to the pulumi registry. Is the initial deploy saving the version in the state file so that it can be referenced in future operations? That latest is used initially still seems like a bug, but at least there is a workaround here.
s
Provider versions are stored in state. Feel free to file an issue in https://github.com/pulumi/pulumi on the initial behavior.
Open in Slack
PreviousNext
Powered by