Hi using pulumi and net and resource resolves has been quite

Question

Hi using pulumi and net and resource resolves has been quite a pain for us currently we are now seeing issues in code that hasn t been changed in ~ 1½ years but all of a sudden some resources are not resolved properly even within output all output tuple we ve even attempted to provoke it by slowing down the code with random sleeps console outputs etc and does indeed seem to work so theres some sort of race condition regarding resource resolving thats not working correctly would love some feedback on the issue

dry-salesmen-32588 · Answer

essentially we are creating a EKSCluster with a custom class extending from ComponentResource within the constructor policies and cluster is created additionally a kubeconfig is also created and its this kubeconfig that is pulling cluster variables and some arns that are null

dry-salesmen-32588 · Answer

reason to why we are revisiting the code is that we needed to upgrade the eks cluster to a supporter cluster version and it seems that some resources are not resolved properly now the original developer has apparently been lucky that the resources were indeed resolved

dry-salesmen-32588 · Answer

we do have plans on rewriting this setup entirely to use the actual ekscluster provider instead of this custom component but for now we are in a jam and need some advice on what we can do here

dry-salesmen-32588 · Answer

constructor + some global vars

dry-salesmen-32588 · Answer

lookuproles function

dry-salesmen-32588 · Answer

the function in which resources must be resolved for generating kubeconfigs

echoing-dinner-19531 · Answer

gt that some resources are not resolved properly now Not totally sure what you mean by this Are you saying that the Outputs from resources are coming back null

dry-salesmen-32588 · Answer

yup

echoing-dinner-19531 · Answer

That the Output lt T gt object itself is null or that the value inside it seen during apply is null

dry-salesmen-32588 · Answer

sometimes its null nullexception and pulumi halts sometimes it goes through

dry-salesmen-32588 · Answer

let me get the last stacktrace sec

dry-salesmen-32588 · Answer

```2022 06 09T07 18 01 5820701Z Diagnostics 2022 06 09T07 18 01 5834051Z pulumi pulumi Stack DevSecOps NgdpPlatform ci 2022 06 09T07 18 01 5835107Z error Running program work src ApplicationPlatform bin Debug netcoreapp3 1 ApplicationPlatform dll failed with an unhandled exception 2022 06 09T07 18 01 5835788Z System NullReferenceException Object reference not set to an instance of an object 2022 06 09T07 18 01 5836668Z at void ApplicationPlatform CaaS Cluster EksCluster CreateKubeConfig string name + ValueTuple lt ClusterCertificateAuthority string string GetRoleResult gt c = gt 5 in work src ApplicationPlatform CaaS Cluster EksCluster cs line 352 2022 06 09T07 18 01 5837530Z at Output lt U gt Pulumi Output lt T gt Apply lt U gt Func lt T U gt func + T t = gt 2022 06 09T07 18 01 5840301Z at async Task lt OutputData lt U gt gt Pulumi Output lt T gt ApplyHelperAsync lt U gt Task lt OutputData lt T gt gt dataTask Func lt T Output lt U gt gt func 2022 06 09T07 18 01 5841455Z error Running program work src ApplicationPlatform bin Debug netcoreapp3 1 ApplicationPlatform dll failed with an unhandled exception 2022 06 09T07 18 01 5842151Z System NullReferenceException Object reference not set to an instance of an object 2022 06 09T07 18 01 5842976Z at void ApplicationPlatform CaaS Cluster EksCluster CreateKubeConfig string name + ValueTuple lt ClusterCertificateAuthority string string gt c = gt 4 in work src ApplicationPlatform CaaS Cluster EksCluster cs line 294 2022 06 09T07 18 01 5843783Z at Output lt U gt Pulumi Output lt T gt Apply lt U gt Func lt T U gt func + T t = gt 2022 06 09T07 18 01 5844350Z at async Task lt OutputData lt U gt gt Pulumi Output lt T gt ApplyHelperAsync lt U gt Task lt OutputData lt T gt gt dataTask Func lt T Output lt U gt gt func 2022 06 09T07 18 01 5844970Z at async Task lt OutputData lt object gt gt Pulumi Output lt T gt Pulumi IOutput GetDataAsync 2022 06 09T07 18 01 5845619Z at async Task lt object gt Pulumi Serialization Serializer SerializeAsync string ctx object prop bool keepResources bool keepOutputValues x 2 2022 06 09T07 18 01 5846550Z at async Task lt RawSerializationResult gt Pulumi Deployment SerializeFilteredPropertiesRawAsync string label IDictionary lt string object gt args Predicate lt string gt acceptKey bool keepResources bool keepOutputValues 2022 06 09T07 18 01 5847615Z at async Task lt SerializationResult gt Pulumi Deployment SerializeFilteredPropertiesAsync string label IDictionary lt string object gt args Predicate lt string gt acceptKey bool keepResources bool keepOutputValues 2022 06 09T07 18 01 5848839Z at async Task lt PrepareResult gt Pulumi Deployment PrepareResourceAsync string label Resource res bool custom bool remote ResourceArgs args ResourceOptions options 2022 06 09T07 18 01 5850085Z at async Task lt string urn string id Struct data ImmutableDictionary lt string ImmutableHashSet lt Resource gt gt dependencies gt Pulumi Deployment RegisterResourceAsync Resource resource bool remote Func lt string Resource gt newDependency ResourceArgs args ResourceOptions options 2022 06 09T07 18 01 5851411Z at async Task lt string urn string id Struct data ImmutableDictionary lt string ImmutableHashSet lt Resource gt gt dependencies gt Pulumi Deployment ReadOrRegisterResourceAsync Resource resource bool remote Func lt string Resource gt newDependency ResourceArgs args ResourceOptions options 2022 06 09T07 18 01 5852655Z at async Task Pulumi Deployment CompleteResourceAsync Resource resource bool remote Func lt string Resource gt newDependency ResourceArgs args ResourceOptions options ImmutableDictionary lt string IOutputCompletionSource gt completionSources 2022 06 09T07 18 01 5853500Z at async Task lt T gt Pulumi Output lt T gt GetValueAsync T whenUnknown 2022 06 09T07 18 01 5854100Z at async Task lt HashSet lt string gt gt Pulumi Deployment GetAllTransitivelyReferencedResourceUrnsAsync HashSet lt Resource gt resources 2022 06 09T07 18 01 5854886Z at async Task lt PrepareResult gt Pulumi Deployment PrepareResourceAsync string label Resource res bool custom bool remote ResourceArgs args ResourceOptions options 2022 06 09T07 18 01 5855982Z at async Task lt string urn string id Struct data ImmutableDictionary lt string ImmutableHashSet lt Resource gt gt dependencies gt Pulumi Deployment RegisterResourceAsync Resource resource bool remote Func lt string Resource gt newDependency ResourceArgs args ResourceOptions options 2022 06 09T07 18 01 5857286Z at async Task lt string urn string id Struct data ImmutableDictionary lt string ImmutableHashSet lt Resource gt gt dependencies gt Pulumi Deployment ReadOrRegisterResourceAsync Resource resource bool remote Func lt string Resource gt newDependency ResourceArgs args ResourceOptions options 2022 06 09T07 18 01 5858554Z at async Task Pulumi Deployment CompleteResourceAsync Resource resource bool remote Func lt string Resource gt newDependency ResourceArgs args ResourceOptions options ImmutableDictionary lt string IOutputCompletionSource gt completionSources SNIP 2022 06 09T07 18 01 5903204Z at async Task lt T gt Pulumi Output lt T gt GetValueAsync T whenUnknown 2022 06 09T07 18 01 5903733Z at async Task lt string gt Pulumi Deployment+EngineLogger TryGetResourceUrnAsync Resource resource 2022 06 09T07 18 01 5904150Z ```

echoing-dinner-19531 · Answer

Odd Why are you wrapping single outputs in `Output All` though That seems pointless

dry-salesmen-32588 · Answer

indeed however it seemed to work for a bit

dry-salesmen-32588 · Answer

alas the error is sporadic

dry-salesmen-32588 · Answer

ie some sort of race condition or we are doing something wrong here

dry-salesmen-32588 · Answer

funny enough while debugging i saw that if i put in a Console WriteLine wahteverdebugtext because i was getting desperate that also seemed to make the error happen less

dry-salesmen-32588 · Answer

ie again pointing to some parallel resolve not going through properly

echoing-dinner-19531 · Answer

well Output All for one output is odd but it should still work

dry-salesmen-32588 · Answer

hence why slightly smiling face

echoing-dinner-19531 · Answer

I d remove the Output All and if you see the error again raise an issue with the stack trace at <http github com pulumi pulumi|github com pulumi pulumi>

echoing-dinner-19531 · Answer

I ll have a pass through the SDK code and see if I can find anything suspect

dry-salesmen-32588 · Answer

running it without the initial output all s does nothing and just brings us back to how the code was written before slightly smiling face

dry-salesmen-32588 · Answer

so in essence they DID DO something but the problem is still there and ultimately didn t solve the problem i guess it worked briefly because of some timing being shifted around abit

dry-salesmen-32588 · Answer

could it somehow be related to the fact that the code is attempting to resolve outputs inside a componentresource

echoing-dinner-19531 · Answer

gt could it somehow be related to the fact that the code is attempting to resolve outputs inside a componentresource I don t think that should matter

dry-salesmen-32588 · Answer

do we have some sort of Promise resolve component1 component2 option to ensure resources are indeed resolved ie wait for component to be populated

dry-salesmen-32588 · Answer

or would it make sense to just make a ugly try catch retry wait loop disappointed

echoing-dinner-19531 · Answer

So I think there s two things here 1 What exactly you mean by resolved 2 If you mean custom or component resources

echoing-dinner-19531 · Answer

1 Resolved to Pulumi just means that it has a value for that property But for example some resource will return their ID and other outputs but they won t actually be ready for use yet So you might try to get pulumi to make a resource then look up it s ID using a cloud sdk and not find it 2 Custom resources should always resolve see above correctly because their properties being set is part of Pulumis core SDK and engine Component resources will only resolve correctly if the programmer who wrote them wrote them correctly their correctness is outside of Pulumis control and there isn t anything we can do to help with that

dry-salesmen-32588 · Answer

so basically the resource is not resolving thats the whole problem

dry-salesmen-32588 · Answer

and yeah we get that the original developer wasn t aware that the resources are actually not being resolved or at least some sort of race condition when accessed

dry-salesmen-32588 · Answer

but it is using pulumis recommended way of writing resources unless theres a specific issue regarding the way we are doing it

dry-salesmen-32588 · Answer

so in the situation where a resource is not resolved do you have a workaround for testing or making sure the resource is indeed resolved before accessing its property

dry-salesmen-32588 · Answer

or is it simply considered a bug and we will have to figure out a way around it ourselves by hacking around it

echoing-dinner-19531 · Answer

gt or is it simply considered a bug and we will have to figure out a way around it ourselves by hacking around it Yup unfortunately I mean the issue is an object property isn t set when you expect it to be set I think you could write a loop to keep trying to read the property until it comes back non null and then use that but I don t think there s much value in trying to write a helper function in the pulumi SDK to do that

dry-salesmen-32588 · Answer

k ill try some hack workaround and if that does indeed work then ill create a ticket with the issue and the workaround fix for someone to dig into

echoing-dinner-19531 · Answer

thumbsup and if do find any instances of properties being null on a Pulumi resource like our aws azure resources do raise a bug on us for it Properties should always be set to a real Output lt T gt object you shouldn t ever be seeing null outputs

dry-salesmen-32588 · Answer

yup a horrible try catch retry loop seems to fix the issue confused

dry-salesmen-32588 · Answer

``` int retrymax=10 int retrycounter=0 bool annoyingresourcehack=true while annoyingresourcehack try UserKubeConfig = Output Tuple certauth endpoint arn Apply c = gt var k8SConfiguration = new K8SConfiguration Clusters = new List lt k8s KubeConfigModels Cluster gt new k8s KubeConfigModels Cluster Name = c Item3 ClusterEndpoint = new ClusterEndpoint Server = c Item2 CertificateAuthorityData = c Item1 Data Users = new List lt User gt new User Name = c Item3 UserCredentials = new UserCredentials ExternalExecution = new ExternalExecution ApiVersion = <http client authentication k8s io v1alpha1|client authentication k8s io v1alpha1> Command = aws Arguments = new region Config Region eks get token cluster name name Contexts = new List lt Context gt new Context Name = c Item3 ContextDetails = new ContextDetails User = c Item3 Cluster = c Item3 ApiVersion = v1 CurrentContext = c Item3 var serializer = new Serializer return serializer Serialize k8SConfiguration annoyingresourcehack=false catch NullReferenceException e if retrycounter==retrymax Console WriteLine retries exhausted while attempting to fetch UserKubeConfig in CreateKubeConfig str throw e Console WriteLine unable to resolve resource for UserKubeConfig in CreateKubeConfig str retrying Thread Sleep 5000 retrycounter++ ``` its ugly and i felt dirty after i wrote this fix but evidently our pipelines can now run without failing disappointed

echoing-dinner-19531 · Answer

That s really odd because that try catch won t catch anything inside an apply which suggests it s `Output Tuple certauth endpoint arn ` which is throwing But also that is a hot loop Apply function don t run synchronously so your probably creating hundreds or thousands of apply objects doing this