https://pulumi.com logo
#kubernetes
Title
# kubernetes
o

orange-policeman-59119

09/09/2021, 8:23 PM
@gorgeous-egg-16927 would it be possible to get a review on https://github.com/pulumi/pulumi-kubernetes/pull/1706/ ?
I had a support ticket with Google Cloud and they were ambiguous about what an "acceptable" rate limit is. (They don't publish what their GKE masters can support.) However
kube-apiserver
has a default
--max-requests-in-flight
to 400 (see here), however even when deploying stacks of 10s to 100s of Kubernetes resources, our Prometheus metrics show that we're well within healthy limits. I think Pulumi's CLI could safely execute many more GETs and even safely make many more mutating requests without any issue.
Without this MR, our production deploys frequently are client-side rate limited. We'd really like to tune the
_QPS
to 50 or 100.
Copy code
Diagnostics:
  pulumi:pulumi:Stack ([redacted]):
    I0909 19:10:50.802281     788 request.go:668] Waited for 1.199010568s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:01.002332     788 request.go:668] Waited for 11.396214694s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:11.201928     788 request.go:668] Waited for 21.593080023s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:21.202128     788 request.go:668] Waited for 31.588109331s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:31.402070     788 request.go:668] Waited for 32.194414591s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:41.601261     788 request.go:668] Waited for 32.194768684s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:11:51.601897     788 request.go:668] Waited for 30.394501934s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:12:01.602219     788 request.go:668] Waited for 12.396325824s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:12:11.801351     788 request.go:668] Waited for 10.8279416s due to client-side throttling, not priority and fairness, request: GET:https://...
    I0909 19:12:21.801661     788 request.go:668] Waited for 3.994815306s due to client-side throttling, not priority and fairness, request: GET:https://...
g

gorgeous-egg-16927

09/10/2021, 4:04 PM
Hey, I’ve been out sick, so I’m trying to catch up on my backlog now. This does look like a good opportunity to tune our defaults, so thanks for starting the work on this! I left a comment on your PR about the approach, and I expect we may bump up the defaults as well.
o

orange-policeman-59119

09/10/2021, 7:29 PM
@gorgeous-egg-16927 sorry about that! hope you're doing well. I'll reply on the GitHub PR
11 Views