following up from

Question

< bored oyster 3147> following up from I did a quick test forcing an error which didn t work as expected and the next run has hung

worried-city-86458 · Answer

This is the forced failure ```Changes Type Name Operation gt pulumi pulumi StackReference pharos aws eks alpha read kubernetes core ServiceAccount kube system aws load balancer controller delete kubernetes cert manager controller issuers delete kubernetes kube system cert manager leaderelection delete kubernetes cert manager cert manager webhook dynamic serving delete kubernetes kube system cert manager cainjector leaderelection delete kubernetes cert manager webhook delete kubernetes cert manager controller orders delete kubernetes cert manager controller clusterissuers delete kubernetes cert manager controller ingress shim delete kubernetes apps Deployment cert manager cert manager delete kubernetes cert manager controller approve cert manager io delete kubernetes cert manager controller clusterissuers delete kubernetes aws load balancer controller rolebinding delete kubernetes aws load balancer webhook delete kubernetes kube system aws load balancer selfsigned issuer delete kubernetes external dns delete kubernetes core Service cert manager cert manager delete kubernetes cert manager webhook subjectaccessreviews delete kubernetes cert manager webhook subjectaccessreviews delete kubernetes core Service alpha internet gateway delete kubernetes cert manager edit delete kubernetes apps Deployment kube system external dns delete kubernetes kube system aws load balancer controller leader election rolebinding delete kubernetes alpha internal gateway delete kubernetes aws load balancer controller role delete kubernetes cert manager controller ingress shim delete kubernetes cert manager cainjector delete kubernetes cert manager cainjector delete kubernetes cert manager webhook delete kubernetes core Service cert manager cert manager webhook delete kubernetes cert manager controller certificates delete kubernetes apps Deployment kube system aws load balancer controller delete kubernetes apps Deployment cert manager cert manager cainjector delete kubernetes cert manager controller challenges delete kubernetes core Service alpha internal gateway delete kubernetes kube system cert manager cainjector leaderelection delete kubernetes core ServiceAccount cert manager cert manager cainjector delete kubernetes core Service kube system aws load balancer webhook service delete kubernetes external dns delete kubernetes cert manager cert manager webhook dynamic serving delete kubernetes cert manager controller certificates delete kubernetes cert manager controller orders delete kubernetes cert manager controller issuers delete kubernetes core ServiceAccount cert manager cert manager webhook delete kubernetes cert manager controller approve cert manager io delete kubernetes kube system aws load balancer serving cert delete kubernetes kube system aws load balancer controller leader election role delete kubernetes kube system cert manager leaderelection delete kubernetes cert manager controller challenges delete kubernetes cert manager controller certificatesigningrequests delete kubernetes cert manager view delete kubernetes core ServiceAccount cert manager cert manager delete kubernetes core ServiceAccount kube system external dns delete kubernetes core Service kube system external dns delete kubernetes cert manager controller certificatesigningrequests delete kubernetes aws load balancer webhook delete kubernetes alpha internet gateway delete kubernetes apps Deployment cert manager cert manager webhook delete kubernetes delete kubernetes delete kubernetes delete Diagnostics pharos k8s alpha pulumi pulumi Stack error Running program D Devel Mps devops gemini pulumi Gemini bin Debug gemini dll failed with an unhandled exception Scriban Syntax ScriptRuntimeException InternalGateway yaml 5 17 error The variable or function `envName` was not found at void Scriban TemplateContext CheckVariableFound ScriptVariable variable bool found at object Scriban TemplateContext GetValue ScriptVariableGlobal variable at object Scriban Syntax ScriptVariableGlobal GetValue TemplateContext context at async ValueTask lt object gt Scriban TemplateContext GetOrSetValueAsync ScriptExpression targetExpression object valueToSet bool setter at async ValueTask lt object gt Scriban TemplateContext GetValueAsync ScriptExpression target at async ValueTask lt object gt Scriban Syntax ScriptVariable EvaluateAsync TemplateContext context at async ValueTask lt object gt Scriban TemplateContext EvaluateAsync ScriptNode scriptNode bool aliasReturnedFunction x 2 at async ValueTask lt object gt Scriban Syntax ScriptExpressionStatement EvaluateAsync TemplateContext context at async ValueTask lt object gt Scriban TemplateContext EvaluateAsync ScriptNode scriptNode bool aliasReturnedFunction x 2 at async ValueTask lt object gt Scriban Syntax ScriptBlockStatement EvaluateAsync TemplateContext context at async ValueTask lt object gt Scriban TemplateContext EvaluateAsync ScriptNode scriptNode bool aliasReturnedFunction x 2 at async ValueTask lt object gt Scriban Syntax ScriptPage EvaluateAsync TemplateContext context at async ValueTask lt object gt Scriban TemplateContext EvaluateAsync ScriptNode scriptNode bool aliasReturnedFunction x 2 at async ValueTask lt object gt Scriban Template EvaluateAndRenderAsync TemplateContext context bool render at async ValueTask lt string gt Scriban Template RenderAsync TemplateContext context at async void Pulumi Deployment+Runner+ lt gt c DisplayClass10 0 lt WhileRunningAsync gt g HandleCompletion|0 +HandleCompletion in sdk dotnet Pulumi Deployment Deployment Runner cs line 137 at async Task lt int gt Pulumi Deployment+Runner WhileRunningAsync in sdk dotnet Pulumi Deployment Deployment Runner cs line 177 Resources delete 61 28 unchanged Duration 13s```

worried-city-86458 · Answer

Preview completed AFAICT

worried-city-86458 · Answer

It still shows a bunch of deletes queued up as a result would that happen if I ran up instead I guess I ll have to try an actual update once I can work out why it now hangs and see what happens

bored-oyster-3147 · Answer

are you using a pre release version

worried-city-86458 · Answer

Yes latest alpha

worried-city-86458 · Answer

Basically I ran preview with forced error then I ran preview again which has hung still spinning 25m later

bored-oyster-3147 · Answer

I would like to verify that the latest alpha includes the changeset that I made

bored-oyster-3147 · Answer

It says it went out 2 hours ago but I don t know what commit it was built from

worried-city-86458 · Answer

I can navigate to source and see your changes

bored-oyster-3147 · Answer

well I m not sure why it would be hanging especially on preview

bored-oyster-3147 · Answer

was this an existing stack Did you by chance already have pending deletes in it

worried-city-86458 · Answer

It was an existing stack that was fully baked All I did was tweak it to force an error

bored-oyster-3147 · Answer

well if you can get a repro let me know

worried-city-86458 · Answer

Still hanging Strange it hasn t timed out

worried-city-86458 · Answer

Had to kill the debug session ```Changes Type Name Operation gt pulumi pulumi StackReference pharos aws eks alpha read Diagnostics pharos k8s alpha pulumi pulumi Stack error transport is closing Resources 21 unchanged Duration 35m27s```

worried-city-86458 · Answer

< bored oyster 3147> debugging into it it seems the exception is never thrown

worried-city-86458 · Answer

i e it returns a preview result

bored-oyster-3147 · Answer

what does `LocalRuntimeService` look like

bored-oyster-3147 · Answer

that implies there was no `CommandException`

worried-city-86458 · Answer

< bored oyster 3147> ` callerContext ExceptionDispatchInfo is null` Want to do a quick interactive session to debug it while I share my screen in slack

bored-oyster-3147 · Answer

I m busy at the moment is that EDI instance null that would cause an exception to not be thrown

bored-oyster-3147 · Answer

do you by chance have anything in your inline program that would cause the exception to not bubble out of it

worried-city-86458 · Answer

No not catching at that scope

bored-oyster-3147 · Answer

but the EDI instance is still null

worried-city-86458 · Answer

`Deployment Runner WhileRunningAsync HandleCompletion` sees the exception which is `Scriban Syntax ScriptRuntimeException` Calls `HandleExceptionAsync` which logs it and returns 32 `Deployment RunInlineAsync` then has `null` `exceptionDispatchInfo` and returns 1 in lambda returns `null` at end `LanguageRuntimeService Run` then returns `new RunResponse `

bored-oyster-3147 · Answer

what is different about this exception

bored-oyster-3147 · Answer

is it thrown in an apply or something

worried-city-86458 · Answer

It does happen inside `Output Create`

bored-oyster-3147 · Answer

and are you using a inline program delegate or the generic TStack

worried-city-86458 · Answer

``` gateways new ConfigGroup internal gateway new ConfigGroupArgs Yaml = RenderTemplate InternalGateway yaml ReadResource new Aws = AwsConfig new ComponentResourceOptions Provider = k8sProvider new ConfigGroup internet gateway new ConfigGroupArgs Yaml = RenderTemplate InternetGateway yaml ReadResource new Aws = AwsConfig new ComponentResourceOptions Provider = k8sProvider ```

worried-city-86458 · Answer

I m deriving from `Pulumi Stack` not directly

worried-city-86458 · Answer

`ConfigGroupArgs Yaml` is `InputList lt string gt `

worried-city-86458 · Answer

`RenderTemplate` returns `Output lt string gt `

bored-oyster-3147 · Answer

so you are using `PulumiFn Create lt TStack gt `

worried-city-86458 · Answer

Yes `PulumiFn Create IServiceProvider serviceProvider Type stackType `

worried-city-86458 · Answer

```var stackName = $ Config Pulumi Organization Name settings Environment ToLower var stackArgs = new InlineProgramArgs info ProjectName stackName PulumiFn Create ServiceProvider info StackType Logger = LoggerFactory CreateLogger lt Pulumi Deployment gt var stack = await LocalWorkspace CreateOrSelectStackAsync stackArgs ```

bored-oyster-3147 · Answer

so not `PulumiFn Create lt TStack gt ` ok just making sure I m looking in the right place

worried-city-86458 · Answer

No the stack type is selected based on the resources to deploy so not using the generic method

bored-oyster-3147 · Answer

OK I have a failing test

worried-city-86458 · Answer

My gut feel is `Pulumi Deployment Runner RunAsync` and `WhileRunningAsync` are swallowing the exception when they should not i e the try catch should be removed or rethrow They do log the exception so maybe not removed

worried-city-86458 · Answer

Then any immediate exceptions in the stack ctor or deferred via outputs should propagate and be captured

bored-oyster-3147 · Answer

yes exceptions in the in flight tasks are being swallowed

worried-city-86458 · Answer

and with the above change implementations of `PulumiFn InvokeAsync` should not rethrow since they are capturing the exception

worried-city-86458 · Answer

Then the captured exception would make it out to `Deployment RunInlineAsync` <https github com pulumi pulumi blob 258fb00bc2ecbd489af6d694a2204468cc7ca729 sdk dotnet Pulumi Deployment Deployment Inline cs L52>

worried-city-86458 · Answer

And finally `LanguageRuntimeService Run` where it would no longer be `null` and so return the bail response <https github com pulumi pulumi blob 258fb00bc2ecbd489af6d694a2204468cc7ca729 sdk dotnet Pulumi Automation Runtime LanguageRuntimeService cs L61>

worried-city-86458 · Answer

The bail response only sets the exception message so the inner exception handler should still be called to log the full exception and rethrow

bored-oyster-3147 · Answer

I went about it a little differently Namely because `IRunner` is used by local programs too and rethrowing in `IRunner` would cause other issues there

worried-city-86458 · Answer

Sounds good I think we understand it either way

bored-oyster-3147 · Answer

Also because the main issue here was the exit code not being threaded through for inline programs At the very least we should ve seen `CommandException` without an inline host exception which was the first thing I fixed Then needed to do some work to capture an aggregate of in flight exceptions so that the explicit exception could bubble up

bored-oyster-3147 · Answer

what a pain in the ass though Me not touching `Deployment Runner` sooner coming back to bite me

bored-oyster-3147 · Answer

Glad you caught that

worried-city-86458 · Answer

FWIW I like your fix LGTM

bored-oyster-3147 · Answer

That PR to fix the swallowed exceptions was merged btw

worried-city-86458 · Answer

Yeah I m already using the latest alpha Much better now