I have a kubernetes cluster with a dozen of deploy...
# kubernetesn
numerous-energy-27817
02/23/2024, 10:08 AMI have a kubernetes cluster with a dozen of deployments and services. I've set dependencies (through inputs-outputs) such that
1. deployments depend on some services (those that the deployment connects to)
2. every service depends on it's matching deployment.
3. every deployment depends on a secret
This way everything works, but create, update and destroy of the cluster is VEEEEERY SLOW
NOw, if I lift the dependencies in (2) during create I see the following
Many (but not all, why?) services appear (as per kubectl get services) and then everything is stuck forever for the secret is never created.
The secret does not depend on anything. Why is it not created?
I have an impression that pulumi simply has a limit on number of resources in "creating state". if this limit is reached pulumi won't create new resources even though they don't have any dependencies and even though those resources waiting for something will wait forever
Does that make sense?
numerous-energy-27817
02/23/2024, 11:45 AMthis limit kinda equals 8 🙂 repeatedly doing "pulumi up" ends up with 8 services waiting until timeout (although there are 10 services that could be waiting)
numerous-energy-27817
02/24/2024, 7:58 PMFixed by setting --parallel 32
Pulumi team, this is an obvious bug.
The part of pulumi command that solves the dependency graph and decides on the order of creating resources does not take implicit dependencies between services and deployments into account. (pulumi implicitly considers a service ready when the backing deployment is ready) But this fact is not taken into account when deciding what in the graph to run first for the whole thing to have any chance to succeed
as I said in initial statement, if I explicitly set dependencies between services and deployments (inputs/poutputs) then it works with the default --parallel of 16
b
billions-xylophone-85957
02/26/2024, 11:25 AMdefault parallel is set here https://github.com/pulumi/pulumi/blob/master/pkg/cmd/pulumi/up.go#L41-L43
n
numerous-energy-27817
03/14/2024, 9:03 AMI believe any program must be able to succeed even with parallel = 1,
If I'm not mistaken parallel is a performance option, not a correctness option
b
billions-xylophone-85957
03/15/2024, 12:23 PMI would make a GH issue, ideally with a small repro