https://pulumi.com logo
#general
Title
# general
b

best-xylophone-83824

02/13/2019, 1:11 PM
I am reading docs I can't find execution model. From the code of examples it seems the moment resource object is created all it's CRUD operations are performed, is it true? Does it mean that in a single pulumi run it is possible lets say create canary deploy and then delete it/update it (lets say scale down)? in other words, can it be an imperative program made of declarative blocks?
👍 1
g

gorgeous-egg-16927

02/13/2019, 4:24 PM
b

best-xylophone-83824

02/13/2019, 4:31 PM
that is exactly 2 resources I checked before asking 🙂 runtime section is more about serializing functions and staged rollout is "ordered" only because it does not know value of the annotation until promise from the canary fires. These kind of tricks of implementing ordering are what terraform is full of and it is enough for some cases, but it is not expressive enough to cover all of them
take that prometheus example for instance. can it be changed so that canary deployment is deleted after main deployment is done? so that state checkpoint never sees canary at all, yet during pulimi run it is declaratively created and then removed?
g

gorgeous-egg-16927

02/13/2019, 4:36 PM
I haven't tried something like that myself, but @white-balloon-205 or @creamy-potato-29402 might have some ideas
b

best-xylophone-83824

02/13/2019, 4:43 PM
Probably having a whole
pulumi up
call on a Stack as a library function would be enough. then we can chain transition between declarative states without falling back to wrapping
pululmi
in bash and that would be killer feature
lets take replicated postgres deploy. either of them can be master, so to update both pods, deploy needs to be smart which one it does first and also do some app-level prep work to minimize downtime during graceful switchover, so it can inspect state of the world and then produce a list of declarative states describing full stack. if they are available as a promises then they can be chained together with set of operations in between. this alone would eliminate need for "operator" pattern which are generally of a very low quality and replace with a set of "operator librarires" to be shared and pulled in as any normal npm package
but ideally every resource should be a promise, not a whole stack. it would allow more concurrency
c

creamy-potato-29402

02/13/2019, 8:47 PM
@best-xylophone-83824 hi Maxim, I think I recognize you from ksonnet community. (I was the original engineer on that)
there isn’t really a way to explicitly do what you describe… you can’t tell pulumi what to do, you can only really tell it what you want. So running a Pulumi program is basically creating a graph of resource operations.
which are then executed by the engine.
As for the “replacing operators” story, we’re looking at that in the background, but I’m not entirely sure yet how this scenario should work
I think for sure it will be useful to chain together things like this, but I’m not sure how to do it and still be declarative.
b

best-xylophone-83824

02/13/2019, 9:05 PM
Yes, I remember you too :) jsonnet/ksonnet is great in describing declarative stuff, but reality is more complicated. I don't know Pulumi internals, but if every resource had .promise method it would cover all the cases I care about
c

creamy-potato-29402

02/13/2019, 9:06 PM
Another context where this comes up is when we’re doing gated deploys based on prometheus checks. We have a (complicated) example: https://github.com/pulumi/examples/tree/master/kubernetes-ts-staged-rollout-with-prometheus
b

best-xylophone-83824

02/13/2019, 9:06 PM
I can then chain multiple modifications of the same resource together in a single Pulumi run. Probably it would need to either do multiple state snapshots or use some "transient" snapshot to feed providers so that they can calculate diff
w

white-balloon-205

02/13/2019, 9:06 PM
FWIW - Every property on a Pulumi resource is of type
Output<T>
which is effectively a promise that also tracks dependency information. This allows you to chain operations in amongst the deployment steps for some scenarios (and there's more we're looking to enable here). See also https://pulumi.io/reference/programming-model.html#outputs.
c

creamy-potato-29402

02/13/2019, 9:09 PM
So there are three parts: canary deployment -> prometheus check -> “the rest of the rollout”
the p8s check depends on canary. The “prod” ring depends on the p8s check.
b

best-xylophone-83824

02/13/2019, 9:09 PM
Yes that Prometheus deploy is what I use as basis for my questions, reason it works is value of annotation is not known so it have to wait for it before creating next declaratively described resource. But it conflates ordering with dependency which are not the same thing
c

creamy-potato-29402

02/13/2019, 9:10 PM
that’s right.
b

best-xylophone-83824

02/13/2019, 9:10 PM
So question comes down to, can Prometheus example be modified so that canary deploy is deleted/scaled down after main deploy is done?
c

creamy-potato-29402

02/13/2019, 9:11 PM
not right now, unfortunately.
b

best-xylophone-83824

02/13/2019, 9:12 PM
If it is yes or Pulumi can be relatively easy adjusted to support it, then complex rollout scenarios are possible all described in the same Pulumi platform. Because right now state of DevOps tools is appalling
c

creamy-potato-29402

02/13/2019, 9:12 PM
So, this is the sort of thing I’m super interested in exploring, but we haven’t even really begun to think about how/whether this fits into the pulumi model…
Totally agree.
That’s why I’m so interested in this example in particular.
tell me more about all the things you’d like to do in a
.promise
is it just scaling down?
b

best-xylophone-83824

02/13/2019, 9:14 PM
There is a problem of expressiveness. There is a big lean into declarative with mantra that declarative is better, but then it reality hits and me finding myself writing tons of bash around declarative tools just to have ordering between declaratively described states
c

creamy-potato-29402

02/13/2019, 9:15 PM
Yeah, I agree. I think k8s historically competes with bash, in that people used to deploy their servers by `scp`’ing a tarball onto a machine, and then `ssh`’ing in to start it. k8s replaces that with a RESTful API, but the bash didn’t go away. It just got moved.
Anyway: super interesting space, something I’d like to see us approach in earnest, but not a thing I feel confident enough to commit product direction to.
b

best-xylophone-83824

02/13/2019, 9:25 PM
Simple example I have right now: updating PostgreSQL master slave replica. To do so I first need to inspect state of that setup: who is master, is replica up to date (abort if not, unless manual override "trust me I know what I am doing" is given), from there can be built a plan for series of transitions: update replica pod, wait for it to catch up, demote master , promote replica, update ex-master. All of these in theory require single change to every resource, so Pulumi should be sufficient, but then in practice I DO want to code abort scenario and in that case error chain should update resources which were updated already in the given Pulumi run
c

creamy-potato-29402

02/13/2019, 9:33 PM
Right, makes sense. So in this situation what you are looking for is, like, a jenkins-esque layer
Some way to use the object model of Pulumi to orchestrate
pulumi up
, and so on
I guess my question is: can this be a layer on top of what we have already, or does it need to be baked into the SDK?
b

best-xylophone-83824

02/13/2019, 9:44 PM
just looking at Typescript examples it feels like having promises on every resource with support multiple updates per run is the right level of abstraction. With power of sharing "recipes" as easy as NPM publish it be awesome. True nirvana would come if then it becomes possible to feed events into this Pulumi program (Azure's Brigate is a step in right direction in this space). Then single Pulumi program can rule whole lifecycle of application: from merge request chatbots, image builds, test runs, arbitrary complex deploys and operator-likefeatures to look after app.
Layer on top would be "stack as a promise" , which would allow everything I am talking about, but probably be too coarse and as a result a bit cumbersome to use.
c

creamy-potato-29402

02/13/2019, 9:49 PM
So, how does this work with the plan step?
One of the nice things about the pulumi model as it exists now is that the plan step is accurate. It tells you exactly what things it will do.
b

best-xylophone-83824

02/13/2019, 9:51 PM
Can I have plan step as object (promise) in typescript program? Many of them for the same stack? If so it will be a good start
w

white-balloon-205

02/13/2019, 9:52 PM
@best-xylophone-83824 Definitely agree with your overall way of thinking about this - much of this is already possible, and everything else you outline is something we want to expand into.
having promises on every resource
We do indeed have this via
Output<T>
(I dropped a note above - but see also https://pulumi.io/reference/programming-model.html#outputs)
multiple updates per run
This is a piece that is not currently supported - we have some thought on how to add it while still maintaining a desired state model.
image builds
FWIW - we do this in many examples today with the
@pulumi/docker
package which combines image build with infrastructure provisioning into a single workflow which can version together.
stack as a promise
I think you are talking about something like this? https://github.com/pulumi/pulumi/issues/2209
b

best-xylophone-83824

02/13/2019, 10:10 PM
Luke, thanks I saw that. It all comes down to be able to update same resource/stack multiple times in a single run. We have acceptance tests were we scale deployment from zero to 1 and then back to zero (preparing PVC with right data before the test ) , it upsets me that I do it in bats (bash test framework :) ) Docker images : I saw it, but I cant express "rebuild this image every time base image is updated" , "rebuild AND deploy this image every time build image is updated", "rebuild image every X days with no cache". That is all next level, I am not talking about feeding events into Pulumi yet, but that is my dream, have a single tool , preferably in typed well supported language to describe all aspects of application lifecycle with the help of battle tested libraries: want images to be signed and security scanned? Wrap image build job into another job which comes from another repo written to by images scannerrl vendor. Want blue rainbow deploys? Here is community written library which takes deployment object and some callbacks (producing app metrics lets say) and then does the rest.
@white-balloon-205, multiple updates per run, is there and issue or Google Doc describing possible approaches? Maybe I can help with coding prototype
h

helpful-holiday-1475

02/15/2019, 2:45 AM
Thanks guys for having this discussion : I asked a related question below (using machine types of K8s clusters as an example) To express recursion, you need turing completeness, which is nowhere in the devops toolbox unless you pysh (python or sh) it yourself