This message was deleted.

Have you seen https://pulumi.io/reference/programming-model.html ? There's a section in there about runtime code. Also check out https://github.com/pulumi/examples/tree/master/kubernetes-ts-staged-rollout-with-prometheus

that is exactly 2 resources I checked before asking 🙂 runtime section is more about serializing functions and staged rollout is "ordered" only because it does not know value of the annotation until promise from the canary fires. These kind of tricks of implementing ordering are what terraform is full of and it is enough for some cases, but it is not expressive enough to cover all of them

I haven't tried something like that myself, but @white-balloon-205 or @creamy-potato-29402 might have some ideas

Probably having a whole

pulumi up

call on a Stack as a library function would be enough. then we can chain transition between declarative states without falling back to wrapping

pululmi

in bash and that would be killer feature

@best-xylophone-83824 hi Maxim, I think I recognize you from ksonnet community. (I was the original engineer on that)

Yes, I remember you too :) jsonnet/ksonnet is great in describing declarative stuff, but reality is more complicated. I don't know Pulumi internals, but if every resource had .promise method it would cover all the cases I care about

Another context where this comes up is when we’re doing gated deploys based on prometheus checks. We have a (complicated) example: https://github.com/pulumi/examples/tree/master/kubernetes-ts-staged-rollout-with-prometheus

I can then chain multiple modifications of the same resource together in a single Pulumi run. Probably it would need to either do multiple state snapshots or use some "transient" snapshot to feed providers so that they can calculate diff

FWIW - Every property on a Pulumi resource is of type

Output<T>

which is effectively a promise that also tracks dependency information. This allows you to chain operations in amongst the deployment steps for some scenarios (and there's more we're looking to enable here). See also https://pulumi.io/reference/programming-model.html#outputs.

So there are three parts: canary deployment -> prometheus check -> “the rest of the rollout”

Yes that Prometheus deploy is what I use as basis for my questions, reason it works is value of annotation is not known so it have to wait for it before creating next declaratively described resource. But it conflates ordering with dependency which are not the same thing

that’s right.

So question comes down to, can Prometheus example be modified so that canary deploy is deleted/scaled down after main deploy is done?

not right now, unfortunately.

If it is yes or Pulumi can be relatively easy adjusted to support it, then complex rollout scenarios are possible all described in the same Pulumi platform. Because right now state of DevOps tools is appalling

So, this is the sort of thing I’m super interested in exploring, but we haven’t even really begun to think about how/whether this fits into the pulumi model…

There is a problem of expressiveness. There is a big lean into declarative with mantra that declarative is better, but then it reality hits and me finding myself writing tons of bash around declarative tools just to have ordering between declaratively described states

Yeah, I agree. I think k8s historically competes with bash, in that people used to deploy their servers by `scp`’ing a tarball onto a machine, and then `ssh`’ing in to start it. k8s replaces that with a RESTful API, but the bash didn’t go away. It just got moved.

Simple example I have right now: updating PostgreSQL master slave replica. To do so I first need to inspect state of that setup: who is master, is replica up to date (abort if not, unless manual override "trust me I know what I am doing" is given), from there can be built a plan for series of transitions: update replica pod, wait for it to catch up, demote master , promote replica, update ex-master. All of these in theory require single change to every resource, so Pulumi should be sufficient, but then in practice I DO want to code abort scenario and in that case error chain should update resources which were updated already in the given Pulumi run

Right, makes sense. So in this situation what you are looking for is, like, a jenkins-esque layer

just looking at Typescript examples it feels like having promises on every resource with support multiple updates per run is the right level of abstraction. With power of sharing "recipes" as easy as NPM publish it be awesome. True nirvana would come if then it becomes possible to feed events into this Pulumi program (Azure's Brigate is a step in right direction in this space). Then single Pulumi program can rule whole lifecycle of application: from merge request chatbots, image builds, test runs, arbitrary complex deploys and operator-likefeatures to look after app.

So, how does this work with the plan step?

Can I have plan step as object (promise) in typescript program? Many of them for the same stack? If so it will be a good start

@best-xylophone-83824 Definitely agree with your overall way of thinking about this - much of this is already possible, and everything else you outline is something we want to expand into.

having promises on every resource

We do indeed have this via

Output<T>

(I dropped a note above - but see also https://pulumi.io/reference/programming-model.html#outputs)

multiple updates per run

This is a piece that is not currently supported - we have some thought on how to add it while still maintaining a desired state model.

image builds

FWIW - we do this in many examples today with the

@pulumi/docker

package which combines image build with infrastructure provisioning into a single workflow which can version together.

stack as a promise

I think you are talking about something like this? https://github.com/pulumi/pulumi/issues/2209

Luke, thanks I saw that. It all comes down to be able to update same resource/stack multiple times in a single run. We have acceptance tests were we scale deployment from zero to 1 and then back to zero (preparing PVC with right data before the test ) , it upsets me that I do it in bats (bash test framework :) ) Docker images : I saw it, but I cant express "rebuild this image every time base image is updated" , "rebuild AND deploy this image every time build image is updated", "rebuild image every X days with no cache". That is all next level, I am not talking about feeding events into Pulumi yet, but that is my dream, have a single tool , preferably in typed well supported language to describe all aspects of application lifecycle with the help of battle tested libraries: want images to be signed and security scanned? Wrap image build job into another job which comes from another repo written to by images scannerrl vendor. Want blue rainbow deploys? Here is community written library which takes deployment object and some callbacks (producing app metrics lets say) and then does the rest.

Thanks guys for having this discussion : I asked a related question below (using machine types of K8s clusters as an example) To express recursion, you need turing completeness, which is nowhere in the devops toolbox unless you pysh (python or sh) it yourself