https://pulumi.com logo
#dotnet
Title
# dotnet
w

worried-city-86458

05/27/2021, 11:49 PM
Say I really want to
await
an
Output<T>
rather than use
Output<T>.Apply
, how should I do that? Any examples?
Are there any other catches like making sure I only call it during the second pass - outside of preview?
b

bored-activity-40468

05/28/2021, 12:13 AM
Stolen from pulumi C# unit test examples.
👍 1
w

worried-city-86458

05/28/2021, 12:25 AM
Interesting. Presumably the
TaskCompletionSource
only completes during up and not preview. So I'd still have to be careful to avoid `await`ing it during preview.
I'd just finished typing up an extension:
Copy code
public static class OutputExtensions
{
    public static Task<T> GetValueAsync<T>(this Output<T> output) =>
        (Task<T>)typeof(Output<T>).GetMethod("GetValueAsync", BindingFlags.NonPublic)!.Invoke(output, Array.Empty<object?>())!;
}
Seems a bit silly either way when this should just be public...
@tall-librarian-49374 let's chat more loosely in here...
What if those outputs never resolve and your await never completes? That sounds dangerous to me...
Excuse my ignorance but how would that be different from doing
Output.Tuple.Apply
to render the template? (Which is a lot more awkward to use, hence I'd much prefer to push the await to Scriban)
t

tall-librarian-49374

05/28/2021, 6:38 AM
The program never blocks on
Apply
. The callback will just never be invoked. An
await
will freeze forever.
w

worried-city-86458

05/28/2021, 6:53 AM
I still don't understand.
Output.Tuple
will still await all the tuple items before calling apply. So if the outputs never resolve, what's the difference between awaiting there vs in a Scriban function? Scriban is async all the way down, so nothing should be "blocking".
t

tall-librarian-49374

05/28/2021, 7:36 AM
Nothing is blocked on
TupleHelperAsync
in that example, it all stays within outputs that know how to handle unknown values etc. I guess I should come up with a repro of a problematic case. It would involve an await in the main function.
Copy code
await Deployment.RunAsync(async () =>
{
    var resourceGroup = new ResourceGroup("my-rg");
    await resourceGroup.Name.GetValueAsync();
    return new Dictionary<string, object?>
    {
        { "test", "this will never run in the initial preview" }
    };
});
w

worried-city-86458

05/28/2021, 9:40 AM
Ah okay, so "naked" await - in the main function - could be dangerous. So if all awaits are "wrapped" within an output then you wouldn't have any other concerns? For my use case the template render is wrapped:
Copy code
return Output.Create(template.RenderAsync(context).AsTask());
t

tall-librarian-49374

05/28/2021, 9:44 AM
Anything that happens in the same function after an await may not run
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Create(Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            await resourceGroup.Name.GetValueAsync();
            return "this will never run either";
        })) }
    };
});
w

worried-city-86458

05/28/2021, 10:15 AM
Does wrapping in an output not help then? I'm confused again.
Surely the tuple helper et al are subject to the same caveats?
t

tall-librarian-49374

05/28/2021, 11:55 AM
If that’s what you mean, this will work:
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Create(Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            return Output.Tuple(resourceGroup.Name, resourceGroup.Location).Apply(v => v.Item1);
        })) }
    };
});
The preview will not hang (it won’t show an output).
w

worried-city-86458

05/28/2021, 6:17 PM
Sorry for sounding dumb but I really need to grok this if I'm to be able to master and extend Pulumi... If I understand correctly, this is all about preview, so if I only run up then this is never a problem. However, if I run preview then I think you're saying that, effectively, it never gets awaited, ever.
Furthermore, in the preview case, tuple above is acting as a "gate" that avoids this problem, flowing either a cached value or unknown? If so, what's the most fundamental building block that I can use to get the same behaviour?
Actually, does
Apply
have the gate in ApplyHelperAsync?
I think where this is going is that I would need to "prune the execution path" upstream of the actual await (and unwrap) for preview to work.
In my case I'd say that must be where I'm creating the output:
Copy code
Output.Create(template.RenderAsync(context).AsTask());
If I understand correctly, what's missing is another building block:
Copy code
Output.Invoke(template.RenderAsync(context).AsTask());
t

tall-librarian-49374

05/28/2021, 7:25 PM
Outputs are like Task’s that may never complete. I think that’s indeed a preview-only problem. We even have OutputCompletionSource. It also has isKnown flag and a shortcut like this. If you can stay await-free, that’s your best bet.
What is
Output.Invoke
?
w

worried-city-86458

05/28/2021, 7:32 PM
It might not be a separate method but maybe
Output.Prune
would be more descriptive. Then again maybe it could be baked into
Output.Create
since it already knows about
Task
. As for what it is, I was thinking about a helper that "prunes task execution" during preview. i.e. if it's got a known value, great, use that, otherwise like
Apply
it should short circuit and return an unknown value.
This would be the gate that bypasses the downstream await and unwrap
It's basically
Apply
semantics but for the initial
Create
Perhaps it looks like this:
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Prune(() => Output.Create(Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            await resourceGroup.Name.GetValueAsync();
            return "this will never run either ... but it won't hang preview";
        }))) }
    };
});
Or if
Prune
chains the
Create
, the indirection is enough:
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Prune(() => Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            await resourceGroup.Name.GetValueAsync();
            return "this will never run either ... but it won't hang preview";
        })) }
    };
});
In which case, maybe it's just another overload of
Create
. But it probably needs to be more explicit that the
Task
will be bypassed during preview.
b

bored-activity-40468

05/28/2021, 7:53 PM
Good stuff, I’ll have to dig into this when home
t

tall-librarian-49374

05/28/2021, 9:55 PM
I’m not sure how Prune would work here… How does it run the resource group constructor but not run the await?
w

worried-city-86458

05/28/2021, 11:41 PM
Bad example - I took your case without looking at it properly. The intention would be to prune where it makes sense, at the "output scope" nearest to the use of await and unwrap. So if you should always run the resource group then the prune happens after that.
My use case is hopefully more of a no brainer:
Copy code
Output.Prune(() => template.RenderAsync(context).AsTask());
Although the more I type prune the more I like create (with indirection):
Copy code
Output.Create(() => template.RenderAsync(context).AsTask());
Hacking around the Pulumi source, leveraging existing
Apply
semantics, I think it's this:
Copy code
public static partial class Output
{
    public static Output<T> Create<T>(Func<Task<T>> func)
        => Create(default(T)!).Apply(_ => func());
}
@tall-librarian-49374 I think I worked it out: https://github.com/pulumi/pulumi/pull/7170
The key that I missed above is that you must use
Apply
on an unknown output.
t

tall-librarian-49374

05/29/2021, 6:08 AM
Q1: In your original example, why can’t you
Output.All
your arguments and only then call
template.RenderAsync
in the apply? Q2: If
Create(default(T)!).Apply(_ => func())
works for you, why do you need to add stuff to Pulumi core?
TBH, I’m reluctant to add
SafeAwaitable
- it’s going to be hard to explain to folks what that is. It also defies the purpose of hiding
GetValueAsync
in
OutputUtilities
.
w

worried-city-86458

05/29/2021, 6:16 AM
A1:
Output.All
and
Output.Tuple
are very clumsy to use in comparison to what I can do otherwise using an anonymous type:
Copy code
var yaml = RenderTemplate("AwsAuth.yaml", ReadResource, new { deployerRoleArn, nodeRoleArn, k8sFullAccessRoleArn, k8sReadOnlyRoleArn });
... and dealing with pulumi outputs in the template A2:
Create(default(T)!).Apply(_ => func())
didn't work - it would hang since the initial output was known ... the
Apply
gate only works when the initial output is unknown
Now that I understand it it's not that hard to explain. In a nutshell,
SafeAwaitable
will skip awaiting the specified awaitable during preview.
It will flow an unknown value as either default (null) during preview or the awaited value during up.
The unit tests demonstrate the above.
If you still don't want to accept the PR, then at the very least can we please make some things public instead of internal so that I can put these extensions / helpers in my own library without needing reflection. 🙏
t

tall-librarian-49374

05/29/2021, 6:26 AM
Hmm… My turn to be puzzled… Why does it even work? Why is the apply callback invoked on an unknown?
w

worried-city-86458

05/29/2021, 6:27 AM
It still gets called during `up`; it's only gated on
preview
(or dry run)
t

tall-librarian-49374

05/29/2021, 6:29 AM
Ah right… that’s somewhat counter-intuitive
make some things public
All you need is
Unknown
, right?
w

worried-city-86458

05/29/2021, 6:32 AM
I think so. That and
OutputUtilities.GetValueAsync
so I can directly await an output in my awaitable.
An
Unknown
is used in a few other places so I could refactor those to use it too.
t

tall-librarian-49374

05/29/2021, 6:37 AM
Unknown
seems safer to me. I’ll take another look on Monday.
👍 1
w

worried-city-86458

05/29/2021, 8:24 PM
I've changed https://github.com/pulumi/pulumi/pull/7170 to specifically address the related issue I'll create another issue and PR for
Output.CreateUnknown
which I now think is a better name