https://pulumi.com logo
Powered by
#dotnet
Title
w

worried-city-86458

05/27/2021, 11:49 PM
Say I really want to 
await
an 
Output<T>
rather than use 
Output<T>.Apply
, how should I do that? Any examples?
Are there any other catches like making sure I only call it during the second pass - outside of preview?
b

bored-activity-40468

05/28/2021, 12:13 AM
Stolen from pulumi C# unit test examples.
Untitled
👍 1
w

worried-city-86458

05/28/2021, 12:25 AM
Interesting. Presumably the 
TaskCompletionSource
only completes during up and not preview. So I'd still have to be careful to avoid `await`ing it during preview.
I'd just finished typing up an extension:
Copy code
public static class OutputExtensions
{
    public static Task<T> GetValueAsync<T>(this Output<T> output) =>
        (Task<T>)typeof(Output<T>).GetMethod("GetValueAsync", BindingFlags.NonPublic)!.Invoke(output, Array.Empty<object?>())!;
}
Seems a bit silly either way when this should just be public...
@tall-librarian-49374 let's chat more loosely in here...
What if those outputs never resolve and your await never completes? That sounds dangerous to me...
Excuse my ignorance but how would that be different from doing 
Output.Tuple.Apply
to render the template? (Which is a lot more awkward to use, hence I'd much prefer to push the await to Scriban)
t

tall-librarian-49374

05/28/2021, 6:38 AM
The program never blocks on 
Apply
. The callback will just never be invoked. An 
await
will freeze forever.
w

worried-city-86458

05/28/2021, 6:53 AM
I still don't understand. 
Output.Tuple
will still await all the tuple items before calling apply. So if the outputs never resolve, what's the difference between awaiting there vs in a Scriban function? Scriban is async all the way down, so nothing should be "blocking".
t

tall-librarian-49374

05/28/2021, 7:36 AM
Nothing is blocked on 
TupleHelperAsync
in that example, it all stays within outputs that know how to handle unknown values etc. I guess I should come up with a repro of a problematic case. It would involve an await in the main function.
Copy code
await Deployment.RunAsync(async () =>
{
    var resourceGroup = new ResourceGroup("my-rg");
    await resourceGroup.Name.GetValueAsync();
    return new Dictionary<string, object?>
    {
        { "test", "this will never run in the initial preview" }
    };
});
w

worried-city-86458

05/28/2021, 9:40 AM
Ah okay, so "naked" await - in the main function - could be dangerous. So if all awaits are "wrapped" within an output then you wouldn't have any other concerns? For my use case the template render is wrapped:
Copy code
return Output.Create(template.RenderAsync(context).AsTask());
t

tall-librarian-49374

05/28/2021, 9:44 AM
Anything that happens in the same function after an await may not run
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Create(Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            await resourceGroup.Name.GetValueAsync();
            return "this will never run either";
        })) }
    };
});
w

worried-city-86458

05/28/2021, 10:15 AM
Does wrapping in an output not help then? I'm confused again.
Surely the tuple helper et al are subject to the same caveats?
t

tall-librarian-49374

05/28/2021, 11:55 AM
If that’s what you mean, this will work:
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Create(Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            return Output.Tuple(resourceGroup.Name, resourceGroup.Location).Apply(v => v.Item1);
        })) }
    };
});
The preview will not hang (it won’t show an output).
w

worried-city-86458

05/28/2021, 6:17 PM
Sorry for sounding dumb but I really need to grok this if I'm to be able to master and extend Pulumi... If I understand correctly, this is all about preview, so if I only run up then this is never a problem. However, if I run preview then I think you're saying that, effectively, it never gets awaited, ever.
Furthermore, in the preview case, tuple above is acting as a "gate" that avoids this problem, flowing either a cached value or unknown? If so, what's the most fundamental building block that I can use to get the same behaviour?
Actually, does 
Apply
have the gate in ApplyHelperAsync?
I think where this is going is that I would need to "prune the execution path" upstream of the actual await (and unwrap) for preview to work.
In my case I'd say that must be where I'm creating the output:
Copy code
Output.Create(template.RenderAsync(context).AsTask());
If I understand correctly, what's missing is another building block:
Copy code
Output.Invoke(template.RenderAsync(context).AsTask());
t

tall-librarian-49374

05/28/2021, 7:25 PM
Outputs are like Task’s that may never complete. I think that’s indeed a preview-only problem. We even have OutputCompletionSource. It also has isKnown flag and a shortcut like this. If you can stay await-free, that’s your best bet.
What is 
Output.Invoke
?
w

worried-city-86458

05/28/2021, 7:32 PM
It might not be a separate method but maybe 
Output.Prune
would be more descriptive. Then again maybe it could be baked into 
Output.Create
since it already knows about 
Task
. As for what it is, I was thinking about a helper that "prunes task execution" during preview. i.e. if it's got a known value, great, use that, otherwise like 
Apply
it should short circuit and return an unknown value.
This would be the gate that bypasses the downstream await and unwrap
It's basically 
Apply
semantics but for the initial 
Create
Perhaps it looks like this:
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Prune(() => Output.Create(Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            await resourceGroup.Name.GetValueAsync();
            return "this will never run either ... but it won't hang preview";
        }))) }
    };
});
Or if 
Prune
chains the 
Create
, the indirection is enough:
Copy code
await Deployment.RunAsync(async () =>
{
    return new Dictionary<string, object?>
    {
        { "test", Output.Prune(() => Task.Run(async () =>
        {
            var resourceGroup = new ResourceGroup("my-rg");
            await resourceGroup.Name.GetValueAsync();
            return "this will never run either ... but it won't hang preview";
        })) }
    };
});
In which case, maybe it's just another overload of 
Create
. But it probably needs to be more explicit that the 
Task
will be bypassed during preview.
b

bored-activity-40468

05/28/2021, 7:53 PM
Good stuff, I’ll have to dig into this when home
t

tall-librarian-49374

05/28/2021, 9:55 PM
I’m not sure how Prune would work here… How does it run the resource group constructor but not run the await?
w

worried-city-86458

05/28/2021, 11:41 PM
Bad example - I took your case without looking at it properly. The intention would be to prune where it makes sense, at the "output scope" nearest to the use of await and unwrap. So if you should always run the resource group then the prune happens after that.
My use case is hopefully more of a no brainer:
Copy code
Output.Prune(() => template.RenderAsync(context).AsTask());
Although the more I type prune the more I like create (with indirection):
Copy code
Output.Create(() => template.RenderAsync(context).AsTask());
Hacking around the Pulumi source, leveraging existing 
Apply
semantics, I think it's this:
Copy code
public static partial class Output
{
    public static Output<T> Create<T>(Func<Task<T>> func)
        => Create(default(T)!).Apply(_ => func());
}
@tall-librarian-49374 I think I worked it out: https://github.com/pulumi/pulumi/pull/7170
The key that I missed above is that you must use 
Apply
on an unknown output.
t

tall-librarian-49374

05/29/2021, 6:08 AM
Q1: In your original example, why can’t you 
Output.All
your arguments and only then call 
template.RenderAsync
in the apply? Q2: If 
Create(default(T)!).Apply(_ => func())
works for you, why do you need to add stuff to Pulumi core?
TBH, I’m reluctant to add 
SafeAwaitable
- it’s going to be hard to explain to folks what that is. It also defies the purpose of hiding 
GetValueAsync
in 
OutputUtilities
.
w

worried-city-86458

05/29/2021, 6:16 AM
A1: 
Output.All
and 
Output.Tuple
are very clumsy to use in comparison to what I can do otherwise using an anonymous type:
Copy code
var yaml = RenderTemplate("AwsAuth.yaml", ReadResource, new { deployerRoleArn, nodeRoleArn, k8sFullAccessRoleArn, k8sReadOnlyRoleArn });
... and dealing with pulumi outputs in the template A2: 
Create(default(T)!).Apply(_ => func())
didn't work - it would hang since the initial output was known ... the 
Apply
gate only works when the initial output is unknown
Now that I understand it it's not that hard to explain. In a nutshell, 
SafeAwaitable
will skip awaiting the specified awaitable during preview.
It will flow an unknown value as either default (null) during preview or the awaited value during up.
The unit tests demonstrate the above.
If you still don't want to accept the PR, then at the very least can we please make some things public instead of internal so that I can put these extensions / helpers in my own library without needing reflection. 🙏
t

tall-librarian-49374

05/29/2021, 6:26 AM
Hmm… My turn to be puzzled… Why does it even work? Why is the apply callback invoked on an unknown?
w

worried-city-86458

05/29/2021, 6:27 AM
It still gets called during `up`; it's only gated on 
preview
(or dry run)
t

tall-librarian-49374

05/29/2021, 6:29 AM
Ah right… that’s somewhat counter-intuitive
make some things public
All you need is 
Unknown
, right?
w

worried-city-86458

05/29/2021, 6:32 AM
I think so. That and 
OutputUtilities.GetValueAsync
so I can directly await an output in my awaitable.
An 
Unknown
is used in a few other places so I could refactor those to use it too.
t

tall-librarian-49374

05/29/2021, 6:37 AM
Unknown
seems safer to me. I’ll take another look on Monday.
👍 1
w

worried-city-86458

05/29/2021, 8:24 PM
I've changed https://github.com/pulumi/pulumi/pull/7170 to specifically address the related issue I'll create another issue and PR for 
Output.CreateUnknown
which I now think is a better name
Powered by