pulumi-community #general

Docs

Join the conversation Join Slack

Channels

general

able-tomato-34262

07/03/2018, 12:55 PM

Huge lag between France and SF... 😄

able-tomato-34262

07/03/2018, 1:23 PM

Sorry for the night spam guys, but is it normal that my

preview

and

update

command using python take ~40 seconds for a 30 lines program ?

able-tomato-34262

07/03/2018, 2:13 PM

Hello, I found that I did my first updates in the wrong region, so I changed the region using

pulumi config set aws:region eu-west-1

( my resources were previously in us-east-1 ) and I expected pulumi to destroy all my resources in

us-east-1

to create them in

eu-west-1

but pulumi seemed to lost track of my resources then, I couldn't do anything, even destroy. Is that the expected behaviour ?

able-tomato-34262

07/03/2018, 2:14 PM

I had to change

aws:region

back to

us-east-1

, destroy, change the config again and then creates them in my new region

important-jackal-88836

07/03/2018, 10:43 PM

if i use a secret config as an input does it end up unencrypted in my checkpoint?

important-jackal-88836

07/03/2018, 11:01 PM

I mean I can see that its unencrypted in

pulumi stack export

important-jackal-88836

07/03/2018, 11:04 PM

how unsafe are secrets in the checkpoint?

white-balloon-205

07/03/2018, 11:05 PM

I just replied in a thread - but a tl;dr is also in the FAQ at https://pulumi.io/reference/faq.html#are-my-secrets-ever-visible.

able-tomato-34262

07/04/2018, 1:39 PM

Hello, I've just had a strange behaviour during an

update

: I tried to update a stack which had no changes and it tried to delete some resource :

$ pulumi update
Previewing update of stack 'prod'
Previewing changes:

   Type  Name  Plan     Info
 
info: no changes required:
      9 resources unchanged

Do you want to perform this update? yes
Updating stack 'prod'
Performing changes:

     Type                 Name               Status                  Info
 *   pulumi:pulumi:Stack  first-pulumi-prod  done                    
 --  └─ aws:rds:Instance  base_c             **deleting failed**     2 errors
 
Diagnostics:
  aws:rds:Instance: base_c
    error: Plan apply failed: deleting urn:pulumi:prod::first-pulumi::aws:rds/instance:Instance::base_c: DB Instance FinalSnapshotIdentifier is required when a final snapshot is required
 
    error: update failed
 
info: no changes required:
      9 resources unchanged

Permalink: <https://app.pulumi.com/jnancel/prod/updates/13>
error: update failed

That's a pretty scary behavior. Any idea what might have happened ?

able-tomato-34262

07/04/2018, 2:37 PM

The only thing I changed is that I added output from some resources

able-tomato-34262

07/04/2018, 2:54 PM

Is it possible to import already created resources into pulumi checkpoint ( like resources created on the Console )?

able-tomato-34262

07/04/2018, 2:57 PM

Is it possible to use an output from a stack in another stack and/or another project ?

quaint-queen-37896

07/06/2018, 5:05 AM

I realise I'm probably in the minority... but having to dig for the powershell install script is annoying 🙂

quaint-queen-37896

07/06/2018, 5:09 AM

Also, is this a bug?

quaint-queen-37896

07/06/2018, 5:09 AM

@quaint-queen-37896 uploaded a file:

image.png▾

quaint-queen-37896

07/06/2018, 6:32 AM

erm.. I've got stuff into a bad state.. and I don't know how to undo it

quaint-queen-37896

07/06/2018, 6:33 AM

need a "nuke the world" and start over option

creamy-potato-29402

07/06/2018, 6:52 AM

what’s the error look like?

quaint-queen-37896

07/06/2018, 6:57 AM

posted one CLI crash to the github issues

quaint-queen-37896

07/06/2018, 6:57 AM

Diagnostics: azure:appservice:Plan: test-appplan error: Plan apply failed: creating urn😛ulumi:pulumi-test-dev::Soda-Digital/pulumi-test::azure:appservice/plan😛lan::test-appplan: web.AppServicePlansClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="" Message=""

quaint-queen-37896

07/06/2018, 7:03 AM

@quaint-queen-37896 uploaded a file:

image.png▾

quaint-queen-37896

07/06/2018, 7:03 AM

clicking the

wtfgarble

option here.. 404's 😕

quaint-queen-37896

07/06/2018, 7:05 AM

annnd now this..

quaint-queen-37896

07/06/2018, 7:05 AM

Do you want to perform this destroy? yes
Destroying stack 'wtfgarble'
Permalink: <https://app.pulumi.com/ctolkien/wtfgarble/updates/8>
error: failed to load resource plugin azure: failed to perform plugin load callback: failed to save snapshot: after mutation of snapshot: child resource urn:pulumi:wtfgarble::Soda-Digital/pulumi-test::azure:core/resourceGroup:ResourceGroup::test-pulumi refers to missing parent urn:pulumi:wtfgarble::Soda-Digital/pulumi-test::pulumi:pulumi:Stack::Soda-Digital/pulumi-test-wtfgarble

quaint-queen-37896

07/06/2018, 7:17 AM

ah ok, that earlier error, re: 400 bad quest.. is because I had an object structured incorrectly

quaint-queen-37896

07/06/2018, 7:18 AM

... still few kinks to get worked out... but this is suuuuper interesting

creamy-potato-29402

07/06/2018, 7:18 AM

interesting. I really only know the kubernetes provider

tall-librarian-49374

07/06/2018, 2:34 PM

When using

cloud.API

, should I

require

my modules inside the

get

function itself? I tried to use

request

imported on top of the files and it failed with a warning not to close on requires... So I do

endpoint.get("/", (req, res) => {
    const request = require('request');
    request(url, ...

helpful-lighter-74534

07/07/2018, 5:56 AM

anyone has an example to create an applicationloadbalancing for EC2 instances?

stocky-spoon-28903

07/08/2018, 1:24 PM

Is there a good example of using the resolved promise of a data source function to modify the control flow of the rest of a program? For example, if I wanted to use the

names.length

result of

aws.getAvailabilityZones()

to pass into something that wants a

number

, and then use the output of that to map to

aws.ec2.Subnet

objects?

Title

stocky-spoon-28903

07/08/2018, 1:24 PM

Is there a good example of using the resolved promise of a data source function to modify the control flow of the rest of a program? For example, if I wanted to use the

names.length

result of

aws.getAvailabilityZones()

to pass into something that wants a

number

, and then use the output of that to map to

aws.ec2.Subnet

objects?

(This is a more generalised version of something @big-piano-35669 and I were discussing via DM, but it is likely of more broad interest).

I see plenty of examples of passing the value of a future into a resource, but can’t spot anything using their results directly for control flow.

big-piano-35669

07/08/2018, 2:54 PM

This is definitely not obvious, so great question. The

getAvailabilityZones

function returns a

Promise<number>

, so you'll need to either

await

its value or use a callback to access it. I prefer

await

, especially if you're using TypeScript, but it does come with one unfortunate "catch": in JS, you cannot

await

at the top level (yet, there is a proposal to enable this). So, this example code will fetch the AZs and simply print them out. From there you can create a subnet-per-AZ, etc.

import * as aws from "@pulumi/aws";
(async () => {
    const zones = (await aws.getAvailabilityZones()).names;
    console.log(`${zones.length} AZs found:`);
    for (const zone of zones) {
        console.log(`\t${zone}`);
    }
})();

If you're creating these in a function, you won't need the cumbersome

(async () => { ... })()

stuff. Instead, it'd just be something like

async function createSubnets() { ... }

. We do have an example of this in the AWS repo, but it's fairly buried and largely only there to serve as a test case: https://github.com/pulumi/pulumi-aws/tree/master/examples/webserver/variants/zones.

stocky-spoon-28903

07/08/2018, 4:30 PM

Does this lead to (for now) wrapping basically the entire program in an async lambda?

At least, if you want to capture the output of something and use it as an input for something else

It sounds like the proposal for top-level await will be particularly useful

big-piano-35669

07/08/2018, 4:37 PM

I wouldn't say there's one answer that works for all cases, however for complex programs we do see this a fair bit. Another pattern is to do something like:

async function main() {
    // all the things happen here
}
main();

We don't await the promise returned from main here, but that's okay, since Node.js keeps the program alive until the event loop quiesces. Another approach is to try to scope the places where you need to do this. For instance, in the subnets case, you might have something like

async function createSubnets() {
    const zones = (await aws.getAvailabilityZones()).names;
    const subnets = [];
    ... create them ...
    return subnets;
}

This can just "push the problem around", because of course the returned subnets array will be wrapped in a promise, but for some cases this is more appropriate. Because our

Input<T>

type accepts promises, doing it like this can sometimes let you achieve a more dataflow style rather than needing to await all over the place.

And yeah, top-level await would be awesome 😊 I gather the challenges there have to do with module loading and asynchrony, especially with exports, ...

stocky-spoon-28903

07/08/2018, 4:39 PM

Right, that’s where I’m currently hitting the problem: exporting the IDs of a bunch of created subnets.

If you use the

async function main()

, exporting anything becomes difficult as far as I can tell

Most of this is just lack of knowledge of typescript idioms though I guess.

Is there a way to create a stack output by allocating something rather than by exporting it from the TS module?

big-piano-35669

07/08/2018, 4:52 PM

A pattern we use in some program is:

async function createResources() {
    // do everything here, including awaiting
    return {
        a: ...,
        b: ...,
    };
}

let outputs = createResources();
export let a = outputs.then(out => out.a);
export let b = outputs.then(out => out.b);

I'm personally not happy with all this ceremony, but as you note, most of this comes from standard JS/TS promises stuff. @lemon-spoon-91807 and @white-balloon-205 had been exploring how to do data sources without needing promises, but I've lost track of where we landed (Node.js makes it hard, by design, to block the message loop).

Is there a way to create a stack output by allocating something rather than by exporting it from the TS module?

Not at the moment, but in principle there's no reason we couldn't. This is all implemented in https://github.com/pulumi/pulumi/blob/015344ab0693ca433d211698eddc6a6a85588bf0/sdk/nodejs/runtime/stack.ts#L27, as called by https://github.com/pulumi/pulumi/blob/015344ab0693ca433d211698eddc6a6a85588bf0/sdk/nodejs/cmd/run/index.ts#L248, by the way, if you were interested in poking around.

stocky-spoon-28903

07/08/2018, 5:01 PM

Aha, that pattern works reasonably well: https://gist.github.com/jen20/2d8451034c0a15858af8561f69a79ac4

👍 1

big-piano-35669

07/08/2018, 5:02 PM

Great! cc @lemon-spoon-91807 @white-balloon-205 @microscopic-florist-22719 as this pattern feels like something we probably want to make more obvious and/or give idiomatic support to

stocky-spoon-28903

07/08/2018, 5:04 PM

If wonder if there couldn’t be something which allows you to export an object from the module directly, and then have the framework take care of separating out each of the values?

That would remove the most objectionable (to me) piece of that pattern, of having to restate each of the outputs to export when the promise is fulfilled

Related: what workflow are people using to test potential changes to the runtime?

big-piano-35669

07/08/2018, 7:21 PM

Make from the root will build and run tests, and install a local dev copy at /opt/pulumi, which you will want on your path, as the language runtime is loaded dynamically.

The READMEs should be up to date on all the steps here. If you're just changing the Node.js language runtime, however, you can get a faster inner loop by just running make from sdk/nodejs/.

If you hit any snags, let me/us know!

stocky-spoon-28903

07/08/2018, 9:04 PM

Thanks. I’m playing with a few ideas for how to improve the situation above now, which means digging further into Typescript than I’ve had to thus far…

https://github.com/tc39/proposal-top-level-await is relevant for anyone following this

big-piano-35669

07/09/2018, 1:15 AM

I tried finding the corresponding GitHub issue in the Typescript repo -- since they usually lead in front on any features likely to make it in -- but came up empty. Since several teammates were involved with Typescript and frequently hang out with those folks, would be good to find out (@white-balloon-205 @lemon-spoon-91807 @bitter-oil-46081).

lemon-spoon-91807

07/09/2018, 1:18 AM

Yes. interesting question

i'm unaware of they're doing anything special in this arena.

the important questions from their perspective (IMO) would be:

1. "how likely is that proposal to get in?" they wouldn't really want to add support for something that ended up not happening.

2. "do we have something suitable we can generate ourselves?"

big-piano-35669

07/09/2018, 1:30 AM

Given the issues and overall velocity around module loading generally, I wouldn't be surprised if this one takes a while to settle. Even Babel seems to be reluctant to pursue it https://github.com/babel/proposals/issues/44. I wish there was a more pleasant way to write code like this. We also had to do this same trickery in our PPC service (our private hosted cloud project).

white-balloon-205

07/09/2018, 3:17 AM

Humorously relevant from 4 years ago when we were standardizing

await

in JavaScript. https://github.com/tc39/ecmascript-asyncawait/issues/9#issuecomment-39000615

👍 1

stocky-spoon-28903

07/09/2018, 7:28 AM

@big-piano-35669 I wonder whether an acceptable answer here would be to allow exporting an

async function(): Promise<any>

from the (user-facing) entry-point module of a Pulumi stack, and if that is what you get back call it from the runtime and use the returned object as the outputs? That way the majority of the ceremony in the pattern above could be removed and replaced with:

interface StackOutputs {
    subnetIds: pulumi.Output<string>[]
}

export default async function(): Promise<StackOutputs> {
    // Do everything here including awaiting
    return {
        subnetIds: someSubnets.map(subnet => subnet.id)
    };
}

big-piano-35669

07/09/2018, 4:47 PM

Ooh that's quite clever! 👍 Do you mind filing a GitHub issue as a suggestion and we can discuss over there?

stocky-spoon-28903

07/09/2018, 5:16 PM

Yup, will do.

👍 2

I’ve opened https://github.com/pulumi/pulumi/issues/1611 to track this.

View count: 16