I had a support ticket with Google Cloud and they were ambiguous about what an "acceptable" rate limit is. (They don't publish what their GKE masters can support.) However

kube-apiserver

has a default

--max-requests-in-flight

to 400 (see here), however even when deploying stacks of 10s to 100s of Kubernetes resources, our Prometheus metrics show that we're well within healthy limits. I think Pulumi's CLI could safely execute many more GETs and even safely make many more mutating requests without any issue.

Without this MR, our production deploys frequently are client-side rate limited. We'd really like to tune the

_QPS

to 50 or 100.

Diagnostics:
  pulumi:pulumi:Stack ([redacted]):
    I0909 19:10:50.802281     788 request.go:668] Waited for 1.199010568s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:01.002332     788 request.go:668] Waited for 11.396214694s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:11.201928     788 request.go:668] Waited for 21.593080023s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:21.202128     788 request.go:668] Waited for 31.588109331s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:31.402070     788 request.go:668] Waited for 32.194414591s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:41.601261     788 request.go:668] Waited for 32.194768684s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:51.601897     788 request.go:668] Waited for 30.394501934s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:12:01.602219     788 request.go:668] Waited for 12.396325824s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:12:11.801351     788 request.go:668] Waited for 10.8279416s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:12:21.801661     788 request.go:668] Waited for 3.994815306s due to client-side throttling, not priority and fairness, request: GET:https://...