https://pulumi.com logo
Powered by
Title
q

quick-garden-9190

03/23/2021, 6:30 PM
I’m in the process of implementing Managed Node Groups and I’m having trouble getting updates to work the way I’d like. With regard to the scaling config, I’m setting a min of 3 hosts and a max of 100. I’d like the cluster-autoscaler to do it’s thing and be able to scale way down if we don’t need the capcity. This becomes a problem considering 
desiredSize
is a required value. This size will clobber whatever the cluster-autoscaler has landed on. I’ve tried using 
ignoreChanges: ['scalingConfig.desiredSize']
but this option seems to be completely ignored. I’ve tried searching through the docs, it seems like this example is missing some key functionality, unless I’m not seeing something obvious.
g

gentle-diamond-70147

03/23/2021, 7:30 PM
Can you elaborate on the behavior you're seeing and the steps to see it? I was doing 
ignoreChanges
for this same use case yesterday and it worked as expected for me.
q

quick-garden-9190

03/23/2021, 8:26 PM
Sure. For example, I’ll instantiate a node group with the following 
scalingConfig
scalingConfig: {
                maxSize: 100,
                minSize: 4,
                desiredSize: 5
            },
Then lets say some time passes and workloads are added to my cluster, the autoscaler will add nodes to satisfy the new requirements. If I were to run another update, even with 
ignoreChanges: ['scalingConfig.desiredSize']
present, the node group will revert back to whatever is defined in 
desiredSize
g

gentle-diamond-70147

03/23/2021, 8:30 PM
Can you share the output from 
preview --diff
that shows this?
q

quick-garden-9190

03/23/2021, 8:38 PM
🤔 Is the purpose of 
ignoreChanges
to ignore any drift that the resources themselves might have made over time? That is, if I were to manually modify the 
desiredSize
even with 
ignoreChanges
in place. Would pulumi update the value because I modified what is declared?
Does that question make sense?
Reason I ask, I’m seeing changes in behavior
Based on the docs I would expect any change I make to 
desiredSize
to be ignored, which is not what I’m seeing
Maybe my tests weren’t properly fleshed out. Let’s say I deploy a node group with a desiredSize of 8 instances. Then I go delete 2 nodes from my cluster. If I run pulumi again it doesn’t attempt to add the 2 nodes back, regardless of whether or not I’m using 
ignoreChanges
This seems to be behavior I haven’t accounted for previously.
Ok, putting that aside for a moment. I have more tests to conduct with desiredSize and the conditions for which it should update. Another scenario I’m trying to account for is what happens when Pulumi needs to delete-replace my node group. Which would happen if I were to change my 
instanceTypes
for example. In that case a new node group would be spun up and I believe 
ignoreChanges
would itself be ignored since it is instantiating the resource. Is it possible for me to lookup how many instances are currently running in an ASG so that I can pass that value as 
desiredSize
?
g

gentle-diamond-70147

03/23/2021, 9:44 PM
hehe, let me catch up...
If values change outside of Pulumi, e.g. through the AWS Console, these will not be reflected in Pulumi's "view of the world" (state file) until you run 
pulumi refresh
or run preview or up with the 
--refresh
flag. This will update Pulumi's state. This would pick up any manual changes you make to 
desiredSize
.
If you set 
ignoreChanges: ["scalingConfig.desiredSize", "scalingConfig.maxSize"]
in your code, then on 
preview
and 
up
, Pulumi will ignore any changes you've made to your code or configuration that is driving these values.
This means if both things happen (you change a value in the console and have 
ignoreChanges
set), your next 
pre
or 
up
should not change these values.
q

quick-garden-9190

03/23/2021, 9:50 PM
I see, thanks for the clarification. with 
scalingConfig.desiredSize
declared as an ignored value, if I modify 
desiredSize
I’m seeing Pulumi recognize the diff and want to update it
Do I need to also include 
scalingConfig.maxSize
for some reason?
g

gentle-diamond-70147

03/23/2021, 9:53 PM
Can you share the 
pulumi pre --diff
output? You should not need to ignore any other properties, unless you see Pulumi trying to update them too and you don't want that to occur.
I've just tested this with my own stack and 
ignoreChanges
is working for all the properties I set.
q

quick-garden-9190

03/23/2021, 9:56 PM
~ launchTemplate: {
              ~ version: "1" => "$Latest"
            }
          ~ scalingConfig : {
              ~ desiredSize: 7 => 9
            }
{
            provider: awsProvider,
            ignoreChanges: ['scalingConfig.desiredSize']
}
g

gentle-diamond-70147

03/23/2021, 10:00 PM
Strange. Would you be able to hop a call tomorrow to troubleshoot this with you? We could talk through some of the other scenarios too.
I work for Pulumi in case that wasn't obvious. 😁
q

quick-garden-9190

03/23/2021, 10:03 PM
That would be great, I’m available at any point tomorrow. I’m in mountain time
g

gentle-diamond-70147

03/23/2021, 10:05 PM
Mind DM'ing me your email?
q

quick-garden-9190

03/23/2021, 10:13 PM
done, thanks
#awsJoin Slack
Powered by