https://pulumi.com logo
Join Slack
Powered by
<@UCU9C7W73> ping
# general
n
@quiet-wolf-18467 ping
so for context, we had a fairly monolithic stack a while back that we had to split to multiple stacks due to deploy times getting far too long (20-30 minis vs. now 1-5 mins). the stack deployed a k8s cluster & all base services on it as well as a couple of separate applications to it
q
What runtime? What's being deployed?
n
splitting the stack from one to multiple has been good overall, but we did lose the static analysis guarantees that existed in the monolithic design (i.e. we know all the services the apps depend on exist because we have them strongly typed). Now we rely on stack references maybe having the data in the format required by the dependant
TS/JS runtime
for what it's worth, we did debug the deployment duration to be caused by Pulumi doing massive amounts of unbatched encrypt/decrypt API calls sequentially, each adding a bit of latency
personally the promise I saw with pulumi is the capability to do full e2e static checking on all of the declarations, but as it is that's not realistically doable & we have to split stacks at some points along the line (currently cluster setup is one stack, each service/deployment in the cluster is their own), I find us in a somewhat similar position as using other tools (although inter-stack static checks are much more versatile & better)
e
we did debug the deployment duration to be caused by Pulumi doing massive amounts of unbatched encrypt/decrypt API calls sequentially, each adding a bit of latency
That's fixed in recent versions of the CLI 🙂
n
i.e. as it is, it'd be hard to do a full deployment of our entire infra to a test cluster, since it's split to multiple stacks which depend on each other in a specific order
That's fixed in recent versions of the CLI
brilliant 👍 that's probably the biggest gripe I've had with pulumi personally so far 🙂
splitting stacks to smaller parts is nice from secret management perspective though (e.g. we don't need cloud API tokens on stacks that only interact with our k8s cluster), but it'd also be nice if that could be solved in monoliths somehow
so in short, I feel the "holy grail" of static analysis is still hard to achieve with Pulumi, or at least was when we were dealing with our monolith (March or so)
e
Because StackReferences themselves aren't typed?
n
I'm personally trying to optimize validating everything before deploying rather than after as much as possible, thus the topic here
Because StackReferences themselves aren't typed?
for one, and we don't actually have the declarative guarantee that the other stack exists & has deployed correctly before deploying the current stack. We can runtime validate it but that's not static analysis
That brings us to the realm of imperative instead of declarative, since stack A depends on the state of stack B
e
If we had something like 
pulumi generate-type-definitions --stack-reference /some/other/stack --language go
that spat out a file with a typed function that did the stack reference based on the current state of the other stack would that help?
n
That would help I suspect, but it wouldn't solve the imperativity which I suspect is the core issue
I'd prefer if the tooling would handle deployment order for us instead of having to organize that ourselves. That's doable in monolithic stacks but not with stack references even if all of the stacks are available locally
Which is how I got to monorepo support
e
Oh right I see, typed and graphed stack references. I think we might have a few things in pipeline that could help with that.
n
Perhaps some way to generate type declarations and automate multi-stack project deployment (akin to yarn workspaces or other monorepo management tools) would be the way to go forward, not quite sure
I just know it's a bit of an issue when deploying multi-stack projects, you lose the declarative nature and static analysis & have to sort of manually figure out the deployment order
Oh right I see, typed and graphed stack references. I think we might have a few things in pipeline that could help with that.
Nice to hear, definitely looking forward to this
Perhaps a pulumi app that deploys other pulumi apps could work but that seems odd 😅 and still lacking cross-stack type checks
I'd look for a way that lets me validate (via static analysis) the entire multi-stack setup, references and all
And that's what we'd have in a CI pipeline
Currently if you preview individual stacks it depends on the state of other stacks it refers to, so even if you make the appropriate changes, preview will fail if you haven't deployed the other stack
e
Yup understood, nothing concrete yet but we do have things in pipeline that will probably help with this 🙂 I like the emphasis on static analysis here, we'll keep that in mind when designing!
👍 1
n
Yeah, personally I think getting static analysis done really well is the ultimate time saver with cloud, given that the feedback loop of actual deployments tends to be quite long. Much prefer having a super good pre-deployment offline validation than runtime validation 😄
Not to mention just ensuring things actually work before pushing them to prod 🙂
2 Views
Open in Slack
PreviousNext
Powered by