This message was deleted.

installing directly:

[Container] 2024/02/14 00:18:27.278435 Running command pulumi plugin install resource aws
E0214 00:18:27.461775     322 plugins.go:430] GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>, try again in 28m49.53822938s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.
error: rate limit exceeded: 403 HTTP error fetching plugin from <https://api.github.com/repos/pulumi/pulumi-aws/releases/latest>

[Container] 2024/02/14 00:18:27.519855 Command did not exit successfully pulumi plugin install resource aws exit status 255

I'm not sure there's much we can do here, other than suggest you supply a

GITHUB_TOKEN

value as suggested to authenticate to GitHub and get around the throttling. Are there concerns/issues with that approach?

I can try that. However, the other CI jobs I have in the same AWS account on the same vpc also report throttling, but they ultimately succeed, consistently. In those jobs, the rate limit isn't accompanied with a 403 error, and the pulumi job doesn't exit with 255 status. In all cases, we are using pulumi v3.105.0.

pulumi preview --stack stage --config-file=Pulumi.stage.yaml
Previewing update (stage):
E0213 22:14:38.142627   11522 plugins.go:430] GitHub rate limit exceeded for <https://api.github.com/repos/pulumi/pulumi-aws/releases/tags/v5.42.0>, try again in 32m8.857380047s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.
[resource plugin aws-5.42.0] installing
@ previewing update......

    pulumi:pulumi:Stack acme-metadata-server-acme-metadata-server-prod-prod  go: downloading <http://github.com/pulumi/pulumi/sdk/v3|github.com/pulumi/pulumi/sdk/v3> v3.78.1
    pulumi:pulumi:Stack acme-metadata-server-acme-metadata-server-prod-prod  go: downloading <http://github.com/pulumi/pulumi-aws/sdk/v5|github.com/pulumi/pulumi-aws/sdk/v5> v5.42.0
...
Resources:
    ~ 3 to update
    +-1 to replace
    4 changes. 65 unchanged


[Container] 2024/02/13 22:15:40.013638 Phase complete: BUILD State: SUCCEEDED

I saw this thread and thought maybe there was some fall back in place that somehow wasn't employed for the failing job.

That's a possibility. This only seems to affect the AWS provider, is that right?

That's all I've seen so far.

What version did you add, if I may ask? (This might help determine what issues, if any, exist with the current fallback mechanism.)

Reproed with both v5.43.0 and v5.42.0

Roger that, thanks for the info.

Still not sure why the other CI jobs work without having to explicitly install the plugin before running the preview, but I did try adding the explicit install without the version on those jobs, and the issue reproduces there as well. 🤷

Got some information from internal teams: “If they don’t supply a version we have to hit the GitHub API to ask what the latest version is. If they’re rate limited then we can’t do that and we’re kinda stuck. We don’t have a way to query

<http://get.pulumi.com|get.pulumi.com>

for versions available.” This explains why adding the explicit version works, but as you pointed out it still doesn’t explain why some environments need the explicit install and others don’t. (Although I do seem to recall that Pulumi YAML needs explicit plugin installs; are you using YAML at all?)

No, we use go for everything.

To me, the logs are consistent with the explanation; it’s hitting GitHub to determine what the latest is and getting throttled, or it’s trying a specific version, getting throttled, and then falling back to

<http://get.pulumi.com|get.pulumi.com>

. And yes, I would expect the version to get pulled from

go.mod

.

I guess what I mean is that I'm surprised that https://api.github.com/repos/pulumi/pulumi-aws/releases/latest is ever used instead of https://api.github.com/repos/pulumi/pulumi-aws/releases/tags/v5.42.0, when the version is available in go.mod.

Yes, agreed. It does seem there is something missing that is causing it to hit the

latest

endpoint when the version is included in

go.mod

.

In any case, thanks for your help. If I find anything, I'll post back. If you think of something, let me know.

Will do!

FYI, I found that if i configured CI to explicitly install the plugin by version and then do pulumi up, everything would work (as already mentioned). Then, if I remove the explicit plugin install from the CI script, pulumi up and pulumi preview will continue to work: it fetches the plugin by version, falling back to the pulumi registry. Is the initial deploy saving the version in the state file so that it can be referenced in future operations? That latest is used initially still seems like a bug, but at least there is a workaround here.

Provider versions are stored in state. Feel free to file an issue in https://github.com/pulumi/pulumi on the initial behavior.