https://pulumi.com logo
Join Slack
Powered by
Hello, We are hitting a bug where, despite the st...
# kubernetes
b
Hello, We are hitting a bug where, despite the stack completing successfully, the status remains 
Ready: False
and stays in 
StackProcessing
. As a result, the workspace pod continuously re-runs, repeatedly syncing outputs and creating a new 
*-stack-outputs
secret on each cycle. With 
resyncFrequencySeconds: 60
, a new secret is created approximately every 60 seconds (we’ve observed 70+ accumulated secrets). Setting 
resyncFrequencySeconds: 0
reduces the frequency but does not fully stop the behavior (something still appears to be re-enqueuing the stack). This is leading to: • Unbounded secret accumulation (etcd pressure) • Unnecessary cloud provider API calls I have also created a GitHub issue with more details: https://github.com/pulumi/pulumi-kubernetes-operator/issues/1163 Is there a known workaround for this issue? Happy to provide additional details, logs, or test a potential fix. Thanks!
s
yeah! so https://github.com/pulumi/pulumi-kubernetes-operator/releases/tag/v2.5.0 shipped an improvement to the ever-rotating Workspace pod. Maybe that release will address your issue. Thank you also for filing your bug. We'll take a look if 2.5.1 doesn't address it!
👀 1
b
Hi Guinevere, thanks for the quick response! We just tested with v2.5.1 and unfortunately the issue persists. After more investigation, we identified the same problem: 
*-stack-outputs
secrets are never garbage collected, each 60s resync creates a brand-new secret and old secrets are never cleaned up. We also found that 
resyncFrequencySeconds: 0
doesn't disable resyncs (per the API docs it defaults to 60s), so there's no way to prevent the accumulation.
Even when a Stack reaches 
Ready: True
with no pending changes, PKO continues to reconcile every 60 seconds indefinitely. Each reconcile runs a no-op 
pulumi up
and creates a new 
*-stack-outputs
Secret but never cleans up the previous one. Over time, these secrets accumulate unbounded. What we expected: Setting 
resyncFrequencySeconds: 0
on the Stack spec would disable periodic resyncs entirely. Instead, PKO treats 
0
as the default (60s), so there's no way to opt out of resyncs.
Hey @shy-arm-32391, Can we switch the outputs secret to use Server-Side Apply? By deriving the secret name from the workspace name, successive updates will patch the same secret in place rather than creating a new one each time. Never mind, It looks like we can use ttl..
s
hi! I've given you some updates on the issue. My hunch is you probably have a secret alwaysResync trigger on your Stack somewhere! Thank you for your patience, and please report back with any findings!
8 Views
Open in Slack
PreviousNext
Powered by