We've had failing (due to timeout) unit tests for a little while. Looking into it now, I've tracked ...
l
We've had failing (due to timeout) unit tests for a little while. Looking into it now, I've tracked it down to
@pulumi/pulumi
3.202.0 -- when I switch to 3.201.0 the tests run fine, and they time out when using 3.202.0, repeatably. Most tests work; I haven't yet tracked down exactly what's different about the 24 that fail versus the 152 that pass. My console logging (in setMocks'
newResource()
) shows that in 3.201.0, all the expect resources are created in `before()`; in 3.202.0, only the top component resource is created, and none of the resources created in their constructors ever get as far as
newResource()
. When dumping the errors, Pulumi is reporting thousands of promises leaked. But I don't read much into that; whenever there's a timeout failure causing a test abort, I see that.
Reviewing the changelog, the most likely change is this one: https://github.com/pulumi/pulumi/pull/20357
But I don't understand the innards of the engine, so take that with a handful of salt.
before()
is not relevant. I can reproduce the problem with all the resources being constructed inside the test.
I presume this is the code that will need to change. What should it look like now?
Hopefully Fraser will spot this in his morning.. I don't want to ping him, it's 3.20am there...
e
Feel free to ping me whenever 🙂 I just pick them up in the morning, I won't get woken up by it. Whats your top resource look like? Most importantly what does it pass to the super constructor for the input args.
Actually more important than the top resource, because sounds like thats the one thats constructing ok. What do the inputs look like for the child resources?
l
Hi! I've determined that all the failures do have a common thread: they're the ones that create our Vpc wrapper component resource as a test dependency (though not necessary in before/beforeAll, I've moved it into the test function while experimenting). The Vpc wrapper is old code, and adds onto awsx.ec2.Vpc, using its older interfaces, with promises rather than apply, to iterate through subnets to build NACLs, security groups and routes.
The actual Vpc object is passed into the constructors of the component resources under test in each case. We prefer to pass IDs, but in these cases there's unknown numbers of subnets, etc. that needs to be handled, so the Vpc wrapper objects is passed.
Maybe there's a race condition somewhere, a pair of things are mutually blocking each other. It's a large amount of code to start commenting out in chunks to figure out what the problem is. At least I can rule out the Vpc peering code that's in there, since I'm not peering in the tests.
I can't give this 100% time today, but I'll keep on at it as best I can and keep this thread updated.
I've commented out almost everything. The smallest case I get get it to happen in is when a component resource constructs an awsx.classic.ec2.Vpc and then refers to its id property later in the constructor. I'll see if I can create a one-file MCVE.
Mocha file that causes the problem. Note that all three of the marked "critical" bits have to be there; remove any of them and it's fine.
I tried reproducing that with simpler resources but couldn't quickly find a simpler pair that reproduced the problem. I'm afraid I've hit my time limit on debugging this for today. It's really weird that setting the parent on both resources is absolutely required....
Oh wow. Just discovered that changing the second
{parent: this}
to either
{parent: undefined}
or even
{parent: this.vpc}
makes the problem go away. It has to be the same parent for both resources.
e
OK thanks that's a lot of information. Nothing stands out as an obvious culprit so I'll have to dig into this a bit.
l
Did this change result in args to a component resource automatically getting stored in the resource? I'm seeing a pile of unexpected updates for my component resources, adding values that are used in the constructor but don't need to be stored.
Hmm... I see passwords that are encrypted in constructors, but Pulumi is wanting to save them into state in plain text because they're in an arg. This isn't ideal.
Yep that's what's happening:
[components/{go,nodejs}] Send component inputs to be saved in state. This brings NodeJS and Go inline with Python behaviour
https://github.com/pulumi/pulumi/pull/20357
That's a big enough change that it deserved a big red warning label, I'm afraid. This is going to require some rewriting in our code.
I don't suppose there's an opt-out for this behaviour is there? Our code assumes that properties are required in order to save something to state, but this change means that args are also saved to state. We're going to need to write new policy.
e
If you need to opt-out you can just change the super call to ComponentConstructor to not pass those args.
but Pulumi is wanting to save them into state in plain text because they're in an arg. This isn't ideal.
That's unexpected. Secrets should still be propagating through args.
That's a big enough change that it deserved a big red warning label, I'm afraid.
Yeh seems its mostly gone unnoticed but has bitten a couple of users badly. Our internal testing had gone fine with only thing causing issues was circular references but we recommend against those anyway.
I'm seeing a pile of unexpected updates for my component
This is actually one of the big wins here, is now the engine can tell that a component is "updated" and so call lifecycle hooks for it.
Looking to add an envvar to opt out of this to help as well. Wouldn't advise depending on it long term, there are good points to saving inputs in state, but should make things like this less urgent panic.
l
If you need to opt-out you can just change the super call to ComponentConstructor to not pass those args.
This is good to know. This may be the quick way to get deployments happening again, then we can fix the couple of poor dev choices we made years ago afterwards.
Secrets should still be propagating through args.
They are. The code that alarmed me involved passing a plaintext secret into a constructor, which then wrapped it in a secret and put it in a property. The plaintext secret wasn't being put into state previously.
> I'm seeing a pile of unexpected updates for my component
This is actually one of the big wins here, is now the engine can tell that a component is "updated" and so call lifecycle hooks for it.
Good stuff. We don't have any of those, but good to know it'll work if we ever create any :)