Is there any correlation between the memory taken up by `pul

Question

Is there any correlation between the memory taken up by `pulumi up` process and the number of objects in the K8s cluster I have a situation where `pulumi up` was crawling to halt on a 8GB worker in CircleCI and when I tried it locally I saw it taking up 20+GB of memory Any tips on troubleshooting this further

colossal-australia-65039 · Answer

how many resources are in your stack and which version of Pulumi are you using one of the more recent minor versions fixed a performance issue causing preview up to be extra slow in case that s one of your symptoms but not sure if the problem solution related to memory usage

hundreds-battery-67030 · Answer

Pulumi CLI version 2 22 0 Pulumi Kubernetes plugin version 2 8 2 We have a few thousand K8s objects

hundreds-battery-67030 · Answer

The net result was that the `pulumi up` process timed out in CI and we had to manually repair the state file

colossal-australia-65039 · Answer

ah that s quite a lot of resources for a single stack not sure if there s a good solution for this to be honest I think it s recommended not to have that many resources in a stack I try to limit mine to the low hundreds at most It s unfortunate because the development experience becomes much more cumbersome when resources are split into smaller stacks but the limits of computational efficiency sort of require it Also there isn t an easy way to split a large stack into smaller stacks You d need to handle the state files manually for that white frowning face

hundreds-battery-67030 · Answer

Correction to my previous response the total number of K8s objects is a few thousand but only a small subset lt 100 is managed via Pulumi gt The net result was that the `pulumi up` process timed out in CI and we had to manually repair the state file This is still the symptom that I m trying to understand better Previously I thought it was the high memory usage that would grind the process to a halt especially in a CI environment But when I mimicked the CI job locally I noticed that the memory usage is reasonably small ~2GB and fairly steady I still see that the `pulumi up` process gets stuck a few minutes after launching By stuck I mean it does not progress in the terminal logs and I do not see any further updates in the cluster state via `kubectl`

hundreds-battery-67030 · Answer

After killing the process after almost an hour it shows the following ```error 2 errors occurred the Kubernetes API server reported that lt redacted replicaset name gt failed to fully initialize or become live Resource operation was cancelled for lt redacted replicaset name gt Attempted to roll forward to new ReplicaSet but minimum number of Pods did not become live``` whereas via `kubectl` I could see the ReplicaSet had achieved minimum availability i e all `Pod`s were ready

colossal-australia-65039 · Answer

sounds to me like it s getting stuck waiting for a ReplicaSet to be ready Are you absolutely positive it is Ready while Pulumi think it s not You re not accidentally checking the wrong namespace or the wrong deployment statefulset etc and the pods aren t flipping between ready and notready I haven t run into an issue with Pulumi incorrectly determining the status of a pod before

hundreds-battery-67030 · Answer

I am fairly certain that I was looking at the right artifacts I am re running `pulumi up` after I ctrl C d it earlier and it might be be stuck again I ll know in a few more minutes This time I m running it with ` profiling` flag to see if it sheds any light I could also DM you the outputs if you d like

colossal-australia-65039 · Answer

sure I m open to DM if you are still seeing the same problem also disclaimer just in case I m not officially affiliated with Pulumi company I m just a random participant in the forum

hundreds-battery-67030 · Answer

Haha thanks for the disclaimer slightly smiling face And I really appreciate you spending time responding to my comments

hundreds-battery-67030 · Answer

I m still seeing the same problem btw this time for a different resource compared to the previous iteration I ll just paste the relevant snippets from my terminal here in case others find it useful From `pulumi up` output ``` 1 2 Waiting for app ReplicaSet be marked available 1 2 Waiting for app ReplicaSet be marked available 0 5 Pods available warning Pod proxy 0nrzpfgd 7d5d8c469d drfd7 containers with unready status proxy updating ``` In a different terminal ```kubectl get po proxy 0nrzpfgd 7d5d8c469d drfd7 NAME READY STATUS RESTARTS AGE proxy 0nrzpfgd 7d5d8c469d drfd7 1 1 Running 0 12m``` I m running `pulumi up` with ` p 1` so I presume it s processing one resource at a time

hundreds-battery-67030 · Answer

When I killed the command it logged the following ``` 1 2 Waiting for app ReplicaSet be marked available 1 2 Waiting for app ReplicaSet be marked available 0 5 Pods available warning Pod proxy 0nrzpfgd 7d5d8c469d drfd7 containers with unready status proxy error 2 errors occurred the Kubernetes API server reported that default proxy 0nrzpfgd failed to fully initialize or become live Resource operation was cancelled for proxy 0nrzpfgd Minimum number of Pods to consider the application live was not attained``` whereas `kubectl` shows the following ```$ kubectl get deploy proxy 0nrzpfgd n default NAME READY UP TO DATE AVAILABLE AGE proxy 0nrzpfgd 10 10 10 10 249d``` This is why I am inclined to believe `pulumi` gets stuck checking the status but I can t tell why or where just yet

colossal-australia-65039 · Answer

hmm this is very weird Your kubectl is also definitely pointing to the same cluster as the CI

colossal-australia-65039 · Answer

actually dumb question it should be or the matching pod name would be ridiculously coincidental

colossal-australia-65039 · Answer

is there any weird diff when you do `pulumi refresh`

hundreds-battery-67030 · Answer

Yeah it s pointing to the same cluster I have 70 resources managed via Pulumi and roughly a dozen of them are K8s `Deployment`s I noticed today that in every `pulumi up` invocation it successfully updates one `Deployment` but gets stuck on the next `Deployment` Other resource types `Secret`s `ConfigMap`s etc do not exhibit this behavior

colossal-australia-65039 · Answer

are there possibly other pods in your cluster that aren t part of that deployment have the same labels are not Ready sort of just grasping at straws now

colossal-australia-65039 · Answer

if the env you re working with is purely experimental I d ask to see what happens when you destroy the resources and recreate them with Pulumi

hundreds-battery-67030 · Answer

This is not an experimental stack unfortunately One thing I d add is this setup worked fine for many months I m pretty sure there is no mismatch in the pods and labels and readiness probes etc Another observation from recent tests although Pulumi CLI gets stuck waiting for a Deployment or a ReplicaSet hard to tell from logs to progress and eventually has to be killed the JSON state file shows the correct updated state of the file when I re run the Pulumi CLI subsequently i e the resource is NOT stuck in `pending operations`

colossal-australia-65039 · Answer

interesting i ran into the same issue as you today when I used a transformation on a kubernetes yaml file to set a deployment replica count I canceled the operation refreshed my state and the next operation succeeded