https://pulumi.com logo
Powered by
s

steep-caravan-65104

04/01/2020, 9:38 AM
Since yesterday (31 March 2020), we have been seeing persistent 429 rate limit errors with the GCS state backend for pulumi. See this error irrespective of the size of pulumi stacks (even on newly created stacks) and even when only a single object from the stack is being created or modified. Also tried 
pulumi up --parallel 1
for single threaded execution, but still see this error.
Copy code
Diagnostics:
  gcp:kms:CryptoKeyIAMBinding (xxx-permissions):
    error: post-step event returned an error: failed to save snapshot: An IO error occurred during the current operation: blob (key ".pulumi/stacks/<stack-name>.json") (code=Unknown): googleapi: Error 429: The rate of change requests to the object <gcs-bucket-name>/.pulumi/stacks/<stack-name>.json exceeds the rate limit. Please reduce the rate of create, update, and delete requests., rateLimitExceeded

  pulumi:pulumi:Stack (<pulumi-project-name>-<stack-name>):
    error: update failed
We may have to give up on using GCS buckets entirely for storing pulumi state. Does anybody know about what could be causing this issue or any workarounds? Thanks. Have created github issue for this too: https://github.com/pulumi/pulumi/issues/4258
👍 1
a

adorable-action-51248

04/01/2020, 9:44 AM
i am also using a GCS bucket for storing pulumi state. i haven’t see this error.
my state is not tiny but also not large.
s

steep-caravan-65104

04/01/2020, 9:48 AM
Maybe the issue is region dependent? Our bucket is in 
asia-east1 (Taiwan)
Could you let us know which region your bucket is in?
a

adorable-action-51248

04/01/2020, 9:50 AM
eu
multi region
s

steep-caravan-65104

04/01/2020, 9:52 AM
Thanks, I'll take a look at replicating that
q

quiet-wolf-18467

04/01/2020, 10:15 AM
I posted this too
s

steep-caravan-65104

04/01/2020, 10:18 AM
Ah, my bad. Didn't spot that before. I'm going to try a multi-region bucket in Asia and report back if it helped. 🤞
q

quiet-wolf-18467

04/01/2020, 10:19 AM
I'll check my bucket config now too
I think mine is single region, europe-west2
a

adorable-action-51248

04/01/2020, 10:43 AM
mmh which version of pulumi are you guys using ?
s

steep-caravan-65104

04/01/2020, 10:44 AM
Copy code
pulumi version
v1.13.1
a

adorable-action-51248

04/01/2020, 10:44 AM
i am still on 
v1.13.0
may be that is a difference ?
s

steep-caravan-65104

04/01/2020, 10:44 AM
Hmm, could be.
I'm now trying a multi-region bucket. It worked once, but will only know for sure if this fixes the issue after running for a while.
👍 1
Also trying 
pulumi up --parallel 1
right now, not sure if that would help
q

quiet-wolf-18467

04/01/2020, 10:53 AM
I've tried 1.12, 1.13, and 1.13.1
😭 1
Parallel flag doesn't help
@steep-caravan-65104 are you using CircleCI by chance?
s

steep-caravan-65104

04/01/2020, 10:57 AM
Nope, Google Cloud Build
q

quiet-wolf-18467

04/01/2020, 10:59 AM
Ah. I get the error when using CircleCI. Doesn't replicate locally
s

steep-caravan-65104

04/01/2020, 11:06 AM
Was seeing it locally too
q

quiet-wolf-18467

04/01/2020, 11:10 AM
Damn
s

steep-caravan-65104

04/01/2020, 12:50 PM
Still getting 429 errors with the multi-region GCS bucket too
Forgot to mention one thing before. Sometimes, the state file gets deleted in this process too when 429 errors are received, which is weird. Have to restore from the backup state files on the bucket in this case.
q

quiet-wolf-18467

04/01/2020, 12:54 PM
Sorry to tag you, @white-balloon-205; but can you let us know if anything has changed with regards to the state file update process?
w

white-balloon-205

04/01/2020, 2:49 PM
Nothing intentionally changed here - but it’s possible we picked up a new version of the gcloud sdk? I see https://github.com/pulumi/pulumi/issues/4258 is open now - we’ll look into that today.
👍 4
q

quiet-wolf-18467

04/01/2020, 2:50 PM
Thank you
c

chilly-laptop-44574

04/02/2020, 4:22 AM
Same here on 
pulumi v1.14.0
and 
Google Cloud SDK 287.0.0
w

white-balloon-205

04/02/2020, 4:29 AM
cc @billowy-army-68599 who has been looking into this.
s

steep-caravan-65104

04/03/2020, 6:26 AM
I've been running pulumi v1.13.0 for a day (have run 5-6 builds so far) and haven't seen this issue again so far. So seems like this issue was indeed introduced with v1.13.1 after all?
👍 1
b

billowy-army-68599

04/03/2020, 3:27 PM
Would you be able to send me verbose logs from both v1.13.0 and v1.13.1 ? I’m unable to repro it @steep-caravan-65104
@quiet-wolf-18467 I saw elsewhere you could repro this on v1.13.0 - is that still the case?
s

steep-caravan-65104

04/04/2020, 8:28 AM
@billowy-army-68599 I've sent you part of the verbose logs from v1.13.1 and v1.13.0 pulumi up runs, hope it helps.
q

quiet-wolf-18467

04/04/2020, 8:33 AM
I've been in vacation the past few days, I'll get our CI logs across on Monday
g

great-byte-67992

04/07/2020, 4:07 AM
I just hit this as well. GCS state, same rate limit error. My stack uses lots of helm charts and i’ve been doing tons of back-to-back up/destroys today. After waiting a little bit I could use pulumi again without the rate limit error.
It seems to happen primarily when using helm charts with tons of k8s resources in them. i’m working around the issue now by using local filestate: 
gsutil cp -r <gs://bucket/.pulumi/stacks> ./.pulumi
and then 
pulumi login file://.
and then using gsutil to re-upload after i’m done.
s

steep-caravan-65104

04/08/2020, 3:01 AM
@great-byte-67992 as mentioned above, you could try Pulumi v1.13.0 where I haven't seen this issue yet. I had passed on logs to @billowy-army-68599 so hopefully will have some resolution soon.
g

great-byte-67992

04/08/2020, 3:02 AM
yeah, i saw that on the github issue. I’ll use 1.13 😄
Thought i’d check in again. using 
@pulumi/pulumi 1.13.0
didn’t resolve the issue. Did you mean to use pulumi CLI 1.13.0?
b

billowy-army-68599

04/09/2020, 12:35 AM
I've yet to confirm that 1.13.0 is not affected, some people say it helps but there's no reason I can see in the code that would confirm that
are you seeing the issue on 
1.13.0
as well?
s

steep-caravan-65104

04/09/2020, 1:11 AM
I've been using 1.13.0 with both docker image and cli for several days now, and haven't seen the issue again so far in either case in hundreds of builds. With 1.13.1, was seeing the issue very frequently. Perhaps the issue is hit more frequently with 1.13.0 in other people's use cases
g

great-byte-67992

04/09/2020, 10:13 PM
I was trying 1.14.0 of the cli with 1.13.0 of the npm package. I’ll try with 1.13.0 for both next week and report back. My bucket is in australia-southeast so that might be why as well.
3 Views
Powered by