https://pulumi.com logo
Powered by
d

dry-keyboard-94795

07/19/2023, 3:50 PM
also very slow to do previews, frequently just hangs (only 65 resources across GCP + k8s). Won't be able to help diagnose today. In the past I've run stacks larger that, while slow, still steadily progress
on this, preview ends up hanging with the last messages being:
Copy code
debug: waiting for quiescence; 13 RPCs outstanding
restricting to 
--parallel 10
lets it get through every time
slowdowns solved with 
pulumi-kubernetes==4.x
(we are using SSA). From ~2 minutes to 10s. Amazing update! Hangs are still around when doing a refresh, needs 
--parallel=5
to progress
update: hangs still every so often with reduced parallel, so needing to disable parallelism entirely for reliability (
--parallel=1
) Any logs I can get you to help with diagnosing + for opening a GH Issue?
g

gorgeous-egg-16927

07/20/2023, 3:37 PM
Hey, glad to hear that pulumi-kubernetes v4 fixed the performance issue for you! The hang sounds like it might be related to https://github.com/pulumi/pulumi/issues/12841 What version of the 
pulumi
package are you using? You can check with 
pulumi about
d

dry-keyboard-94795

07/20/2023, 4:33 PM
3.75
I've noticed a couple of dependency issues when using k8s ConfigFile, where other resources will begin creation before everything nested under the ConfigFile is finished. Could be related, but with it being intermittent, tough to pin down
If I recall correctly, it was the webhook deployment resource from the cert-manager manifest that wasn't wait for. Only noticed as it caused an issue elsewhere when setting up a ClusterIssuer
g

gorgeous-egg-16927

07/20/2023, 4:38 PM
The ConfigFile ordering issue is tracked in https://github.com/pulumi/pulumi-kubernetes/issues/1833
d

dry-keyboard-94795

07/20/2023, 4:40 PM
For now, we'll just disable parallism. It isn't really any slower for our usecase
g

gorgeous-egg-16927

07/20/2023, 4:42 PM
Sounds good. We are planning to address 1833 in the near future, hopefully this quarter.
d

dry-keyboard-94795

07/20/2023, 4:44 PM
This is all at the preview phase btw, not sure how much dependsOn matters there
a

adorable-computer-41765

07/20/2023, 5:21 PM
Are you using ComponentResources?
d

dry-keyboard-94795

07/20/2023, 5:22 PM
Yes
a

adorable-computer-41765

07/20/2023, 5:23 PM
I have seen this historically where there is a loop somewhere, any way you can share the code?
it happens when the component creates a loop thru referencing itself while depending on itself
d

dry-keyboard-94795

07/20/2023, 5:38 PM
Unfortunately not. Do you have an example of what these loops would look like?
I'm not at a computer atm, so I'll test tomorrow (7pm uk time now) if reducing our ConfigFile use fixed the parallel issue. Should at least narrow down the issue for replication
a

adorable-computer-41765

07/20/2023, 5:51 PM
You can sort of see them in this PR: https://github.com/pulumi/pulumi/pull/12855
d

dry-keyboard-94795

07/20/2023, 6:06 PM
Hmm, I did add depends_on=config_file_resource ontop of parent=config_file_resource for a bunch of resources. Basically representing that a k8s Secret is directly related to a cert-manager deployment.
a

adorable-computer-41765

07/20/2023, 6:07 PM
Yep, that is most likely the problem.
d

dry-keyboard-94795

07/20/2023, 6:11 PM
Do you know if typescript has less of these issues? I was considering switching in the past, as to get type hints in python, it's overly verbose
a

adorable-computer-41765

07/20/2023, 6:13 PM
I don't, I think it will have the same. Especially on that.
I would just parent the rest of the resources to self, and depends_on=[config_file_resource]
that should clear it all up
d

dry-keyboard-94795

07/20/2023, 6:23 PM
Cool, thanks for the help. In hindsight, I misused parent= there 😂
with both changes (reduced ConfigFile to CRDs only, and parent=configFile), unbound parallel seems happy now. Thanks!
just tried to replicate the suspected cyclic issue using the CRDs manifest; unfortunately can't make it happen. will try with a fresh project shortly
a

adorable-computer-41765

07/20/2023, 7:23 PM
cool, if its a small set of resources it may not trigger except at like a 25% hit rate.
I debugged the original issue, so that made it tough.
d

dry-keyboard-94795

07/20/2023, 7:35 PM
Haha, yeah. Though if it is the ConfigFile issue, that part is limited to the cert-manager manifest + 2 other resources at the top-level. So not too complex to whittle down.
this replicates the issue most of the time: https://gist.github.com/antdking/b8ec51a5eb24d1321a1e7163e4b61cde having a resource implicitly depend on another resource that depends on its parent. replicated with just a ComponentResource, doesn't need you to run 
up
, just 
preview
from scratch will do it
I assume the expected behaviour is for Pulumi to shout at the user for trying to depend on a parent?
a

adorable-computer-41765

07/20/2023, 8:31 PM
I am trying to raise an issue for that exact question internally
I think on preview we should tell you not to do it
d

dry-keyboard-94795

07/20/2023, 8:32 PM
nice. Will try and simplify this further to use purely ComponentResources, just in case it's something to do with k8s
a

adorable-computer-41765

07/20/2023, 8:36 PM
I mean that is what I hope we get out of it. That is not what it will do now
d

dry-keyboard-94795

07/20/2023, 8:38 PM
yep, I meant I was simplifying it further so there's a better test case to submit
a

adorable-computer-41765

07/20/2023, 8:39 PM
Thanks Anthony, no need though. I can take it from here or you can just use the gist you have now
d

dry-keyboard-94795

07/20/2023, 8:41 PM
brilliant, thanks. Want me to submit a bug report on github, or are you happy with just raising internally?
a

adorable-computer-41765

07/20/2023, 8:42 PM
If you want to report it, that way you are notified I will follow up internally
d

dry-keyboard-94795

07/20/2023, 9:25 PM
2 Views
Powered by