@white-balloon-205 It seems like the

gcp

Cluster

resource is unable to scale up an already existing cluster by changing its

nodeCount

or maybe I’m missing something ?

Type                                 Name                                       Status                   Info
     pulumi:pulumi:Stack                  REDACTED
     └─ REDACTED:Cluster  REDACTED-cluster
 +-     └─ gcp:container:Cluster          REDACTED                   **replacing failed**    [diff: ~initialNodeCount]; 1 error

Diagnostics:
  gcp:container:Cluster (REDACTED):
    error: Plan apply failed: googleapi: Error 409: Already exists: projects/REDACTED/zones/REDACTED/clusters/REDACTED-cluster., alreadyExists

is the node count the only thing that's changed? I remember there being a lot of limitations on what can be changed in a cluster via the api/terraform, which I think is one of the reasons I always create the cluster and any node pools separately these days

@abundant-airplane-93796 Yes, I only wanted to add a node to the cluster

in truth I haven't done it via pulumi yet, but I did run into similar issues back when using terraform itself, and even the google deployment manager. might be unrelated, but what I was hitting was some limitation in how google's api allows changes to a cluster

sounds like related to me since it’s only a wrapper of the terraform provider under the hood. That’s kind of a problem if we are not able to scale up/down a cluster’s nodes even with native google’s api. I’ll need to dig on this one to be sure I won’t hit a wall later on.

are you creating the cluster + the node pool in a single resource, or creating a cluster resouce, and attaching nodepool resources to it?

right now yes, I’m provisioning a

Cluster

resource (in pulumi terms) with all related configurations cluster and nodepool in it. Honestly I’m still new to this, so not sure if what I’m doing is the right way to do it. From what you are saying I understand I could create an “empty” cluster and attach nodepools to it later on ? Which could maybe be more flexible and avoid the error I’m encountering right now ?

yup, exactly that. similar to the "Usage with an empty default pool" example given in the terraform docs here https://www.terraform.io/docs/providers/google/r/container_node_pool.html

indeed, I was not aware of this capability, that will be very helpul. I will try to figure out how to configure this with the pulumi provider

This is what we are doing in our code. No default node pool, build via separate function

@faint-motherboard-95438 we usually create clusters with https://www.terraform.io/docs/providers/google/r/container_cluster.html#remove_default_node_pool set to true (or the pulumi equivalent) and create separate pools then.

thanks @orange-tailor-85423 and @glamorous-printer-66548 I found out this option afterward and supposed that was the way to go but good to have both your confirmation on this matter 🙂 Indeed it works better this way.

the GKE apis don’t allow changing labels of an existing nodepool, so either way (UI / terraform / pulumi) changing a label means delete + recreate. The problem you’re running into is probably https://github.com/pulumi/pulumi/issues/1620 . Basically currently for most resources, when pulumi determines it has to recreate a resource, it will first create the new resource before deleting the old one (that is done to reduce downtime). Since you probably specified a fixed

name

for your nodepool this strategy is running into a conflict. The suggested resolution for now is not to specify a

name

in the nodepool resource, but instead let pulumi auto-generate a name based on the pulumi resource id (for nodepools pulumi will generate a name using

<resource_id>-<some_random_suffix>

). This way pulumi is able to create a new nodepool before deleting the old one in the case of changes which require a recreation (e.g. changing nodepool labels).

@glamorous-printer-66548 oh ok, gotcha, thanks. Makes sense. So, sorry if that’s a dumb question, but if the provider deletes the node in order to make the changes, what happens to the services/pods in this node ? Will they be preserved/recreated along the way without any downtime or should I have to take care of this myself and if yes what is the right way to do it ?

well when GKE deletes a nodepool it first puts it ‘cordons’ / ‘drains’ it’s nodes by applying a taint to them, which prevents scheduling of further pods onto this node. After being a minute or so cordoned the node gets deleted altogether. In the meantime the scheduler might reschedule / move the pods onto the new nodepool. I honestly can’t tell you 100% sure whether there will be any downtime in between as the time it takes to reschedule your pods depends on a couple of factors e.g. how big your images are (the new node will have to redownload the images) but in most cases if there’s any downtime it should be fairly short (i.e. a couple of seconds). If no downtime, not even a few seconds is acceptable to you, you should probably test out the behaviour in detail with your apps.

Sounds crystal clear, thanks a lot for such explanation and for the link, I still need to learn a lot about all of this, but missing time. I’ll try that right away and see how it goes

yeah it should. that’s weird - never seen this before even though I’ve already done lots of updates like changing node count. Can u share ur code, especially the cluster config? Looks like the error you’re getting is on the cluster resource and not on the pool resource which is kinda weird too.

I did a refresh (which updated a lot of things, that’s weird since I didn’t change anything in between) and after a new update the problem was gone. Really weird. I’ll try with a full new install to see if I can reproduce