Hello everyone, is there a pattern to create and update an infrastructure ? I have a Pulumi.yaml file which describes my infra (security group, nodepool, cluster), it’s working fine and allow me to create the infra seamlessly. But upgrading the cluster is more complex as manual intervention is needed: • creation of a new nodepool • draining the node of the previous nodepool • Deleting the previous nodepool playing with yaml only (Pulumi.yaml and Pulumi.STACK.yaml) does not seem to be possible for that purpose. Any best practice to follow ? I guess using one of the language supported by Pulumi is the best approach to automate the upgrade, right ?

Why do you need a manual intervention here? The process of draining and deleting a nodepool should be handled by your cluster management. In the case of managed Kubernetes (which it sounds like you're using based on the keywords you mention) you don't have to worry about this at all, if you delete a nodepool the workloads will be shifted to other nodes. What you describe is the typical default behavior when a property changes that requires re-creation of a resource: The new resource is created, and the old resource is subsequently deleted. It's possible to change this through the deleteBeforeReplace resource option.

Yes, I get this but I was not sure about the way to specify this in the Pulumi.yaml / Pulumi.STACK.yaml files. I mean, if I need to upgrade the cluster, creating a new nodepool, and removing the old one (for that stack only), how this should be done ?

You edit your program so that it includes the new nodepool and doesn't contain the old one, and then run

pulumi up

. If you just want to change something about the nodepool (e.g., machine type) you simply change this particular argument and Pulumi will know whether this requires recreation of the nodepool (in which case it will, by default, make the new one, then delete the old one) or can be done by changing the existing nodepool.

Thanks, that makes a lot of sense. I could then specify the name of the nodepool in the config of my Pulumi.STACK.yaml, right ? I’m using the Exoscale cloud provider

The logical names of Pulumi resources should not depend on configuration or outputs. They should be fixed or generated within your program. Otherwise, you risk replacement or loss of resources (and data!) if the dynamic value changes.

Sorry, yes I was talking about the name of the nodepool as it appears in the cloud provider, not the logical name 👍 I will test this

Per https://www.pulumi.com/registry/packages/exoscale/api-docs/sksnodepool/#inputs you can change the name of a nodepool without replacing it. There's a small symbol next to the inputs that will trigger a replacement, which are the

clusterId

and the

zone

.

Hum, so this is not really what I need. When upgrading the cluster, the id and zone will not change. I guess I should create a new nodepool and delete the old one in a more manual way though

Why do you want to delete the nodepool, though? 🤔 Isn't it a good thing that you can just keep and update your nodepool, rather than waiting for it to be recreated? If you really want to replace the nodepool instead of doing an in-place update you can change its logical name in Pulumi.

Hi @modern-zebra-45309 in fact currently (working with the cloud provider cli) I do not replace the nodepool when upgrading the cluster. Instead I scale the nodepool up so it brings new version of the nodes. For instance, once scaled up, I can have 3 nodes with version 1.29.7 and 3 nodes with version 1.30.3, all in the same nodepool. Then I drain the 1.29.7 nodes and delete them. From what I understand, this can not easily be done using Pulumi, this is why I was wondering if creating a new nodepool and removing the old one in an automated way could be done instead.

Resource options are defined in the Pulumi program, not as part of the stack

the version is set in the SKScluster. Once this version is upgraded each new node (when scaling the nodepool) will get that version

I think you want to set it up in a way that if you update the cluster's version through Pulumi, the nodepools are also updated

yes, when I change the version config in the stack, I’d like to trigger the upgrade of the controlplane (this is working) and also trigger the upgrade of the worker node.

Yes, but if you use replaceOnChanges, it will 🙂

Can I set this in my Pulumi.yaml file ? 🙂

In the Pulumi.yaml file. Let me go find the proper syntax for you, the example in the docs does not work for YAML but just because the CRD they use does not work

cool, thanks

Thanks a lot, I’ll try it

This will happen automatically. When you replace a Pulumi resource and do not set

deleteBeforeReplace

to

True

, the new nodepool will be created first and then the old nodepool will be deleted. This will allow your workloads to shift, it's a very common pattern with Kubernetes clusters. You can trust Kubernetes to handle this re-scheduling for you.

I have tested your approach and it’s working fine.

resources:
  my-resource:
    type: does:not/exist
    properties:
      name: this-is-the-name-with-${cluster.version}
    options:
      replaceOnChanges:
        - name

A new nodepool is created and the previous one is deleted. Also, the workload is migrated to the new nodepool as it’s Kubernetes job to do that part. But (sorry there is a “but” 🙂 ), the old nodepool is deleted just after the new nodepool is created. Then Kubernetes takes a few tens of seconds before deciding to move the workloads. So just after the old nodepool is deleted, the workload is not running anymore (interuption of service), we have to wait for kubernetes to detect the nodes are not there and then to reschedule the workloads on the new node. The proper way to do that would be to have both nodepools in parallel, then to drain the old nodes, then make sure everything is correctly rescheduled, then to delete the old nodepool.

I think this is something that you should be able to configure on the Kubernetes level, see https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/

I think I need to find a way (without relying on the configuration of the cluster, as some params might not be available on all cloud providers) to wait for the workloads to be scheduled on the new nodes before removing the old one 🤔

I've not worked with Kubernetes on Exoscale before but I think it's worth digging into what you can do on a Kubernetes level. You don't want to have service interruptions just because a node is terminated, which can happen anytime. Trying to solve this from the outside seems like a hack/anti-pattern to me. If you switch to a non-YAML flavor of Pulumi, you can use the pattern I linked above (https://gist.github.com/metral/48a576680208d1c9961c37c5b1f0025e or https://gist.github.com/lukehoban/fd0355ed5b82386bd89c0ffe2a3c916a) to wait for Kubernetes resources to become available.

in fact the issue is not because a node is terminated, you’r right this can happen all the time. It’s more that all the nodes (previous nodepool) are terminated at the same time, thus leaving the workload as is. Kubernetes will do its job correctly, but it will take much more time than if the darin was done correctly first 🤔

I see, so you'd need a way to implement a delay or health check, similar to a CloudFormation CreationPolicy to only start removing the old nodepool once the new nodepool is fully available. I don't think this exists in Pulumi.

In fact I would need a way to remove the old nodepool once the drain of its nodes is correctly done so we are sure the workload are now running on the new nodepool.

I think the EKS example uses the approach I mentioned initially: Add a second logical nodegroup in your Pulumi program, run pulumi up, migrate the workloads, remove the original logical nodegroup from your Pulumi program, run pulumi up again.

I though about that but what if I have several stacks (dev, qa, prod) and only want to upgrade the dev cluster ? If I change the program it will be available to each stacks, right ?

Only if you actually deploy the stack with "pulumi up." You can evolve the program underlying each stack separately.

In fact I’d like to have a program (or YAML) and only manage the config of each stack so they use the same base program

Yes, which is how it should be, but you're now mixing in operational deployment/maintenance concerns into your infrastructure code, which should be handled by the provider

In fact, as I saw this upgrade working fine in terraform for EKS I thought this could be easily replicated with Pulumi for Exoscale. But there might be an automated upgrade (taking into account the drain) which is not implemented in the cloud provider side I guess. Do you think each provider maintain its Pulumi library ?

I have not seen the problem you face with EKS. I'm pretty sure (although I have not checked) that node group replacements work smoothly there, at least when everything is appropriately configured.

Thanks, you right I will check that on Exoscale side 👍 BTW, thanks a lot for you help, I understood many things in my Pulumi journey 😀

You're welcome, I'm happy if I can help, it's usually a great learning opportunity for me as well 🙂

Hey @modern-zebra-45309 I think I will go with one of your recommendations, using a single project per environment. Thus my dev cluster will have its own project so I can easily create an additional node pool and drain the old one when it’s there (without impacting the other environment / stack). I wanted to kinda use the same program definition (yaml in my case) for each stack, but that will be too complicated for the upgrade process

Note that if you'd switch from YAML to a programming language like Python, you could re-use parts of your code between programs. You could have a joint cluster setup that you import into your environment-specific programs.