https://pulumi.com
Join Slack
Hello AWS Pulumi people. Is there a way to rate-limit the AWS api calls made by the Pulumi provider...
# aws
c
Hello AWS Pulumi people. Is there a way to rate-limit the AWS api calls made by the Pulumi provider? We are randomly hit with "Throttling Exception: Rate exceeded" errors when updating AWS SES receipt rules. Digging into the documentation it appears that this API is limited to 1 request per second. This is an intermittent problem; AWS does not always enforce this limit. They do, however, seem to enforce this limit whenever we have a critical, time-sensitive operation to complete. 😉 Anyone run into this problem and might have a solution? https://docs.aws.amazon.com/ses/latest/dg/quotas.html#quotas-email-general
image.png
e
We're looking at adding configurable retry options soon, which will allow you to catch throttling exceptions and retry them. Might also be worth asking in the aws provider repo if it makes sense for the provider to always auto-retry throttle errors.
👌 1
l
In the interim I could imagine a hack where you use before hooks to implement a queue in your Pulumi program, assuming that your Pulumi programs are the only source of requests...
👋 1
But this is a hack
c
Thanks @echoing-dinner-19531 - I will do that. I have a feeling this is not the only AWS API that has a low throttle rate.
@lively-crayon-44649 Eeeeeeeenteresting! I imagine I would set up a ~1 second "sleep" timer in the beforeCreate, beforeUpdate and beforeDelete hooks. The next question is would I need to disable pulumi parallelism to ensure the API calls are executed serially?
l
I don't think so -- there will only be one instance of your program running at once, so all hooks will go back into the same language host
So (and again, this is horrible, and probably not a good idea) you could imagine something like
c
Thanks @lively-crayon-44649 - we are going to try that route
l
Copy code
// Making this up, should probably think about whether this works/is data-race-free etc.
let locked = false

const x = new Resource("x", { ... }, {
  hooks: {
    beforeCreate: [() => {
      while (locked) { await new Promise(resolve => setTimeout(resolve, 1_000)) }     
      locked = true
    }],
    afterCreate: [() => { locked = false }],
  },
})
^ This is probably not going to work ever but maybe some better version of the idea might
I'm more of a specialist in introducing race conditions rather than eliminating them unfortunately
😅 1
And yea, sleeps in the before are probably simpler/good enough
c
I will implement this later this week and report back... mid-week holidaytomorrow :D
Thanks @lively-crayon-44649 - adding a sleep timer to the resource hooks worked like a charm:
Copy code
...
from time import sleep
...
def sleep_hook(args: pulumiResourceHookArgs):
  sleep(1)
...
  _ = aws.ses.ReceiptRule(
        ...
        opts=pulumi.ResourceOptions(
          hooks=pulumi.ResourceHookBinding(
            before_create=[sleep_hook],
            before_update=[sleep_hook]
          )
@echoing-dinner-19531 Added a feature req to 
pulumi-aws
thx for the suggestion https://github.com/pulumi/pulumi-aws/issues/5939
l
Nice! I'm torn between "don't let perfect be the enemy of good" and "the thing that should not be", but glad it's working 😄
c
hah - it's not an ugly solution. Thanks sir
Open in Slack
PreviousNext
Powered by

Pulumi Community

No matter how you like to participate in developer communities, Pulumi wants to meet you there. If you want to meet other Pulumi users to share use-cases and best practices, contribute code or documentation, see us at an event, or just tell a story about something cool you did with Pulumi, you are part of our community.

Powered by