https://pulumi.com logo
Join Slack
Powered by
also very slow to do previews, frequently just han...
# general
d
also very slow to do previews, frequently just hangs (only 65 resources across GCP + k8s). Won't be able to help diagnose today. In the past I've run stacks larger that, while slow, still steadily progress
on this, preview ends up hanging with the last messages being:
Copy code
debug: waiting for quiescence; 13 RPCs outstanding
restricting to 
--parallel 10
lets it get through every time
slowdowns solved with 
pulumi-kubernetes==4.x
(we are using SSA). From ~2 minutes to 10s. Amazing update! Hangs are still around when doing a refresh, needs 
--parallel=5
to progress
update: hangs still every so often with reduced parallel, so needing to disable parallelism entirely for reliability (
--parallel=1
) Any logs I can get you to help with diagnosing + for opening a GH Issue?
g
Hey, glad to hear that pulumi-kubernetes v4 fixed the performance issue for you! The hang sounds like it might be related to https://github.com/pulumi/pulumi/issues/12841 What version of the 
pulumi
package are you using? You can check with 
pulumi about
d
3.75
I've noticed a couple of dependency issues when using k8s ConfigFile, where other resources will begin creation before everything nested under the ConfigFile is finished. Could be related, but with it being intermittent, tough to pin down
If I recall correctly, it was the webhook deployment resource from the cert-manager manifest that wasn't wait for. Only noticed as it caused an issue elsewhere when setting up a ClusterIssuer
g
The ConfigFile ordering issue is tracked in https://github.com/pulumi/pulumi-kubernetes/issues/1833
d
For now, we'll just disable parallism. It isn't really any slower for our usecase
g
Sounds good. We are planning to address 1833 in the near future, hopefully this quarter.
d
This is all at the preview phase btw, not sure how much dependsOn matters there
a
Are you using ComponentResources?
d
Yes
a
I have seen this historically where there is a loop somewhere, any way you can share the code?
it happens when the component creates a loop thru referencing itself while depending on itself
d
Unfortunately not. Do you have an example of what these loops would look like?
I'm not at a computer atm, so I'll test tomorrow (7pm uk time now) if reducing our ConfigFile use fixed the parallel issue. Should at least narrow down the issue for replication
a
You can sort of see them in this PR: https://github.com/pulumi/pulumi/pull/12855
d
Hmm, I did add depends_on=config_file_resource ontop of parent=config_file_resource for a bunch of resources. Basically representing that a k8s Secret is directly related to a cert-manager deployment.
a
Yep, that is most likely the problem.
d
Do you know if typescript has less of these issues? I was considering switching in the past, as to get type hints in python, it's overly verbose
a
I don't, I think it will have the same. Especially on that.
I would just parent the rest of the resources to self, and depends_on=[config_file_resource]
that should clear it all up
d
Cool, thanks for the help. In hindsight, I misused parent= there 😂
with both changes (reduced ConfigFile to CRDs only, and parent=configFile), unbound parallel seems happy now. Thanks!
just tried to replicate the suspected cyclic issue using the CRDs manifest; unfortunately can't make it happen. will try with a fresh project shortly
a
cool, if its a small set of resources it may not trigger except at like a 25% hit rate.
I debugged the original issue, so that made it tough.
d
Haha, yeah. Though if it is the ConfigFile issue, that part is limited to the cert-manager manifest + 2 other resources at the top-level. So not too complex to whittle down.
this replicates the issue most of the time: https://gist.github.com/antdking/b8ec51a5eb24d1321a1e7163e4b61cde having a resource implicitly depend on another resource that depends on its parent. replicated with just a ComponentResource, doesn't need you to run 
up
, just 
preview
from scratch will do it
I assume the expected behaviour is for Pulumi to shout at the user for trying to depend on a parent?
a
I am trying to raise an issue for that exact question internally
I think on preview we should tell you not to do it
d
nice. Will try and simplify this further to use purely ComponentResources, just in case it's something to do with k8s
a
I mean that is what I hope we get out of it. That is not what it will do now
d
yep, I meant I was simplifying it further so there's a better test case to submit
a
Thanks Anthony, no need though. I can take it from here or you can just use the gist you have now
d
brilliant, thanks. Want me to submit a bug report on github, or are you happy with just raising internally?
a
If you want to report it, that way you are notified I will follow up internally
d
2 Views
Open in Slack
PreviousNext
Powered by