Hi everyone! I'm still new to Pulumi so I apologise if my question is too straight forward or already answered before here. I have an stack where my machines are already pre-existent in ec2. On each machine there are some web services running using docker containers. I want to manage these containers using Pulumi so I went ahead and used the docker provider with ssh access to the ec2 machines. Now the problem I'm facing is that these services should be gradually rolled out upon any changes. Right now if change the definition and do

pulumi up

all containers get deleted and then pulumi start creating new ones. This is obviously bad in production, this means my service is instantly down. Instead I want to stop one, create it, stop the next one, craete it, etc.. Now I figured out some way to create the machines gradually by waiting on outputs from each resource and providing it as input to the next, etc. But still all my containers get destroyed at the same time. Next thing I tried is marking the containers

retainOnDelete: true

and then add a pulumi Command which deleted the container before creating it. Now the problem is this command doesn't change when the container changes so it is executed only ones. Is there a better way to approach this? Is my line of thinking even in the right direction? How do you do roll out strategies with pulumi when using docker containers or in general?

Use the dependsOn property of the resource. https://www.pulumi.com/docs/iac/concepts/options/dependson/

@magnificent-eve-10499 let's say I have container1 and container2. I want to make sure the update goes like this: 1. container1 is deleted 2. container1 is running 3. container2 is deleted 4. contianer2 is running How does dependsOn here make this happen?

I see. I was thinking you were saying that it was 1. container1 is running 2. container2 is created 3. container1 is deleted dependsOn container2 If the container is in a new cluster on step 2 how is it still "container1"?

No clusters here, I think you confused my question with Hans'. Let's even ignore the cluster side, imagine this is on the same machine. I just want to replace container1 and container2 gradually by allowing only 1 being down at a time

haha! yes I did. sorry about that. so make container2 dependOn container1. in that case container2 will not be touched until container1 is back up and running.

oh wow, I completely got this wrong, you are right

Awesome, glad I could help.

basically the setup is 1. delete container1 2. create container1 3. wait for container1 to respond on /health with true 4. delete container2 5. create container2

What is your cloud provider?

these containers are deployed using the pulumi.Docker provider but the instances live in aws

do you use route53 for the healthcheck?

no, it is some custom logic, it is a bit complicated, the thing is container one starts running quickly but it has to sync some state from another place and I want to check when it synced fully then I can go and stop container2

is the health check registered as a custom health check with the EC2 instance? if so, then I would expect that the dependsOn would not fire until the healthCheck passes.

it is not, there are multiple containers running on this instance, not just one so I can't associate it with the ec2 instance unfortunately

I wonder if this would help then. It appears they are using tags to record the health result. you could make the dependency on a tag that has been updated. https://aws.amazon.com/blogs/compute/how-to-create-custom-health-checks-for-your-amazon-ec2-auto-scaling-fleet/

I'm thinking if I can do a more generic solution using a custom resource, dynamic provider and dependsOn. Let's say I define a custom HealthCheck resource, implement a dynamic provider which does some http call and custom logic based on the resource inputs and make my dependency be: customHealthCheck2 dependsOn container2 dependsOn customHealthCheck1 dependsOn container1

yeah, that makes sense.

when container1 is updated, it would trigger the customHealthCheck1 update, right?

No, it would not. At least not how I understand it.

hm, so this dependency from above

customHealthCheck2 dependsOn container2 dependsOn customHealthCheck1 dependsOn container1

won't work meaning that if only container1 and container2 change, the healthchecks won't trigger?

they will trigger but there will be no "diff" between the current and previous state of the healthcheck. Unless you modify some value pulumi is tracking that is used in the healthCheck it wont plan any changes to the healthcheck so you will not get an updated status check. Dont think of the status check like you do other resources. It is different in that there isn't anything to track until it refreshes from the stack. So in order to use an updated status you will either need to attach the healthcheck to something in the ec2 instance,

get

the state of some other (untracked) resource in Azure that will give you what you need to know about the status, or use pulumi command to query your endpoint for a status and then act on that.

Would the pulumi command work diferently here if I replace the healthcheck with a command? I guess not, same would apply, right?

I think there are 3 directions you could go, but if you want to use this locally you will need to use the command.local.command so that you can do something like a

curl

that will work both on and of cloud.

if I use local command it won't get triggered if the container changes? Let's say I have command1 (which is some curl health check) and container1. command1 dependsOn container1. If I update container1 then command1 won't run?

hold on I will show you how I do it.

Got it! So you can trigger the command based on some Output. Yeah, I should figure out what Output I can use which would change any time a docker.Container definition changes

so you might do something like this. This should wait until pulumi believes that container2 is up and running

is there some way to get an output which is like some consistent hash of the whole resource or would only change when the resource changes or something like that so that I can use for the triggers. I see you use this systemData.lastModifiedAt but I can't find a similar output in the docker.Container object