does anybody experience slowness of pulumi runs Simple run w Pulumi Community #general

does anybody experience slowness of pulumi runs? S...

best-xylophone-83824

09/02/2019, 1:05 PM

does anybody experience slowness of pulumi runs? Simple run with no refresh, only a single stack output variable added, with no resources unchanged per plan has been running for 10 minutes by now... I'll try to gather more debug info

tall-librarian-49374

09/02/2019, 1:08 PM

What do you see during those 10 minutes? Which OS are you on? Which provider?

best-xylophone-83824

09/02/2019, 1:10 PM

It just showed full output and then hangs, for 10 mins ,then prints:

Copy code

Resources:
    159 unchanged

Duration: 10m20s
Permalink: <https://app.pulumi.com/Nakhoda/gcp-gke/gke01-london/updates/129>

we don't use

--refresh

and there were no changes apart from added output variable, so it never hit provider (or it shouldn't at least the way I understand they way pulumi works).

tall-librarian-49374

09/02/2019, 1:13 PM

Interesting. Is it just once or does it happen every time?

best-xylophone-83824

09/02/2019, 1:18 PM

it is reproducible, I just reran same job on a pipeline, zero changes this time , yet it's been 4 minutes still and looks very slow.it trickles "output" objects one by one once a minutes or so. once finishes I'll rerun same commit but from my dev machine to rule out our CI setup

white-balloon-205

09/02/2019, 7:54 PM

Could you run with

—logtostderr -v=9

and see if there are any hints of what is taking so long?

best-xylophone-83824

09/03/2019, 9:41 AM

ha, it exceeded CI job output limit 🙂 if I don't set

--logtostderr

which file does it log to?

best-xylophone-83824

09/04/2019, 11:39 AM

@tall-librarian-49374, checked output, there is a

Marked old state snapshot as done

record for each resource, which seems to be taking 2.3-2.5 seconds each. we have 165 resources so far, so it quickly adds up and our no-op

pulumi up

end up running for > 10 minutes

tall-librarian-49374

09/04/2019, 11:41 AM

Aha! Thanks, I'll ask around.

tall-librarian-49374

09/04/2019, 11:42 AM

Could you file an issue for that?

best-xylophone-83824

09/04/2019, 11:45 AM

doing it now

👍 1

best-xylophone-83824

09/04/2019, 11:50 AM

https://github.com/pulumi/pulumi/issues/3178

tall-librarian-49374

09/04/2019, 11:53 AM

thank you!

best-xylophone-83824

09/04/2019, 4:49 PM

@white-balloon-205, @creamy-potato-29402 , this slowness makes us unproductive, can you think of a workaround we can apply to speed it up? maybe convert to local backend do a full run and convert back ? anything would help

white-balloon-205

09/04/2019, 4:50 PM

Looking into it now. Could you share any more of the detailed logs?

best-xylophone-83824

09/04/2019, 4:50 PM

@white-balloon-205, can send you full logs over email

white-balloon-205

09/04/2019, 4:50 PM

Also - what is the stack name in question?

white-balloon-205

09/04/2019, 4:50 PM

I'm at luke@pulumi.com