Is there a good example of using the resolved prom...
# general
s
Is there a good example of using the resolved promise of a data source function to modify the control flow of the rest of a program? For example, if I wanted to use the
names.length
result of
aws.getAvailabilityZones()
to pass into something that wants a
number
, and then use the output of that to map to
aws.ec2.Subnet
objects?
(This is a more generalised version of something @big-piano-35669 and I were discussing via DM, but it is likely of more broad interest).
I see plenty of examples of passing the value of a future into a resource, but can’t spot anything using their results directly for control flow.
b
This is definitely not obvious, so great question. The
getAvailabilityZones
function returns a
Promise<number>
, so you'll need to either
await
its value or use a callback to access it. I prefer
await
, especially if you're using TypeScript, but it does come with one unfortunate "catch": in JS, you cannot
await
at the top level (yet, there is a proposal to enable this). So, this example code will fetch the AZs and simply print them out. From there you can create a subnet-per-AZ, etc.
Copy code
import * as aws from "@pulumi/aws";
(async () => {
    const zones = (await aws.getAvailabilityZones()).names;
    console.log(`${zones.length} AZs found:`);
    for (const zone of zones) {
        console.log(`\t${zone}`);
    }
})();
If you're creating these in a function, you won't need the cumbersome
(async () => { ... })()
stuff. Instead, it'd just be something like
async function createSubnets() { ... }
. We do have an example of this in the AWS repo, but it's fairly buried and largely only there to serve as a test case: https://github.com/pulumi/pulumi-aws/tree/master/examples/webserver/variants/zones.
s
Does this lead to (for now) wrapping basically the entire program in an async lambda?
At least, if you want to capture the output of something and use it as an input for something else
It sounds like the proposal for top-level await will be particularly useful
b
I wouldn't say there's one answer that works for all cases, however for complex programs we do see this a fair bit. Another pattern is to do something like:
Copy code
async function main() {
    // all the things happen here
}
main();
We don't await the promise returned from main here, but that's okay, since Node.js keeps the program alive until the event loop quiesces. Another approach is to try to scope the places where you need to do this. For instance, in the subnets case, you might have something like
Copy code
async function createSubnets() {
    const zones = (await aws.getAvailabilityZones()).names;
    const subnets = [];
    ... create them ...
    return subnets;
}
This can just "push the problem around", because of course the returned subnets array will be wrapped in a promise, but for some cases this is more appropriate. Because our
Input<T>
type accepts promises, doing it like this can sometimes let you achieve a more dataflow style rather than needing to await all over the place.
And yeah, top-level await would be awesome 😊 I gather the challenges there have to do with module loading and asynchrony, especially with exports, ...
s
Right, that’s where I’m currently hitting the problem: exporting the IDs of a bunch of created subnets.
If you use the
async function main()
, exporting anything becomes difficult as far as I can tell
Most of this is just lack of knowledge of typescript idioms though I guess.
Is there a way to create a stack output by allocating something rather than by exporting it from the TS module?
b
A pattern we use in some program is:
Copy code
async function createResources() {
    // do everything here, including awaiting
    return {
        a: ...,
        b: ...,
    };
}

let outputs = createResources();
export let a = outputs.then(out => out.a);
export let b = outputs.then(out => out.b);
I'm personally not happy with all this ceremony, but as you note, most of this comes from standard JS/TS promises stuff. @lemon-spoon-91807 and @white-balloon-205 had been exploring how to do data sources without needing promises, but I've lost track of where we landed (Node.js makes it hard, by design, to block the message loop).
Is there a way to create a stack output by allocating something rather than by exporting it from the TS module?
Not at the moment, but in principle there's no reason we couldn't. This is all implemented in https://github.com/pulumi/pulumi/blob/015344ab0693ca433d211698eddc6a6a85588bf0/sdk/nodejs/runtime/stack.ts#L27, as called by https://github.com/pulumi/pulumi/blob/015344ab0693ca433d211698eddc6a6a85588bf0/sdk/nodejs/cmd/run/index.ts#L248, by the way, if you were interested in poking around.
s
👍 1
b
Great! cc @lemon-spoon-91807 @white-balloon-205 @microscopic-florist-22719 as this pattern feels like something we probably want to make more obvious and/or give idiomatic support to
s
If wonder if there couldn’t be something which allows you to export an object from the module directly, and then have the framework take care of separating out each of the values?
That would remove the most objectionable (to me) piece of that pattern, of having to restate each of the outputs to export when the promise is fulfilled
Related: what workflow are people using to test potential changes to the runtime?
b
Make from the root will build and run tests, and install a local dev copy at /opt/pulumi, which you will want on your path, as the language runtime is loaded dynamically.
The READMEs should be up to date on all the steps here. If you're just changing the Node.js language runtime, however, you can get a faster inner loop by just running make from sdk/nodejs/.
If you hit any snags, let me/us know!
s
Thanks. I’m playing with a few ideas for how to improve the situation above now, which means digging further into Typescript than I’ve had to thus far…
https://github.com/tc39/proposal-top-level-await is relevant for anyone following this
b
I tried finding the corresponding GitHub issue in the Typescript repo -- since they usually lead in front on any features likely to make it in -- but came up empty. Since several teammates were involved with Typescript and frequently hang out with those folks, would be good to find out (@white-balloon-205 @lemon-spoon-91807 @bitter-oil-46081).
l
Yes. interesting question
i'm unaware of they're doing anything special in this arena.
the important questions from their perspective (IMO) would be:
1. "how likely is that proposal to get in?" they wouldn't really want to add support for something that ended up not happening.
2. "do we have something suitable we can generate ourselves?"
b
Given the issues and overall velocity around module loading generally, I wouldn't be surprised if this one takes a while to settle. Even Babel seems to be reluctant to pursue it https://github.com/babel/proposals/issues/44. I wish there was a more pleasant way to write code like this. We also had to do this same trickery in our PPC service (our private hosted cloud project).
w
Humorously relevant from 4 years ago when we were standardizing
await
in JavaScript. https://github.com/tc39/ecmascript-asyncawait/issues/9#issuecomment-39000615
👍 1
s
@big-piano-35669 I wonder whether an acceptable answer here would be to allow exporting an
async function(): Promise<any>
from the (user-facing) entry-point module of a Pulumi stack, and if that is what you get back call it from the runtime and use the returned object as the outputs? That way the majority of the ceremony in the pattern above could be removed and replaced with:
Copy code
interface StackOutputs {
    subnetIds: pulumi.Output<string>[]
}

export default async function(): Promise<StackOutputs> {
    // Do everything here including awaiting
    return {
        subnetIds: someSubnets.map(subnet => subnet.id)
    };
}
b
Ooh that's quite clever! 👍 Do you mind filing a GitHub issue as a suggestion and we can discuss over there?
s
Yup, will do.
👍 2
418 Views