Moving it from #general to #kubernetes Does anyone know an example of installing and configuring kubernetes "cluster-autoscaler" to use ManagedNodeGroup with pulumi/eks and pulumi/kubernetesx?

I've done this with

pulumi/kubernetes

, not

kubernetesx

, and I don't have a ready example. Are you looking for code, or do you just have questions?

This is on my test branch so may disappear at some stage, and I'm not using pulumi/eks or pulumi/kubernetesx, but this may still be of interest: • EksStack.cs#L206 • K8sStack.cs#L567 (helm charts) • K8sStack.cs#L554 (helm releases) Ideally I'd patch the node group asg tags for cluster autoscaler using a dynamic provider, but that's still not supported in dotnet/c#.

@worried-city-86458 you can probably use the

aws.ec2.Tag

resource to do this, similar mechanism to this: https://github.com/jaxxstorm/pulumi-examples/blob/main/typescript/aws/eks_subnet_tags/index.ts#L51-L57

@worried-city-86458 is my understanding here correct - 1. Set 2 tags on the asg group -

"<http://k8s.io/cluster-autoscaler/node-template/label/role|k8s.io/cluster-autoscaler/node-template/label/role>" - "<nodeGroup.Name>"
"<http://k8s.io/cluster-autoscaler/node-template/taint/role|k8s.io/cluster-autoscaler/node-template/taint/role>" - "NoSchedule"

2. Create a new IAM role for autoscaler using

ClusterAutoscalerPolicy.json

3. Deploy

cluster-autoscaler

helm chart with appropriate settings and passing the RoleArn to RBAC setting on the chart values.

@billowy-army-68599 Ah yes, I'm not sure why I didn't use that instead, I'll have to take another look. Also fwiw, you don't need to tag your subnets for the eks cluster any more.

yeah I wrote that yonks ago and just kept it around mainly in case I needed something like it in future 😄

You do need to tag subnets for aws load balancer controller though so that example will be useful to others using pulumi/eks

So are the only two steps is create a role with irsa policy and deploy the chart?

It depends. Read https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup carefully.

<http://k8s.io/cluster-autoscaler/enabled|k8s.io/cluster-autoscaler/enabled>

tag is applied automatically by managed node group iirc However, if you're applying labels or taints to the node group (like I am here), then you should also apply hints via asg tags, i.e. step 1.

looks great! sometimes inside an Apply isn't the worst option

@worried-city-86458 @billowy-army-68599 Thank you for your help. I have been able to put together first working cut of code to setup autocluster on EKS using managedNodeGroup in Typescript -

// Setup Kubernetes Autoscaler

const clusterOidcProvider = cluster.core.oidcProvider;
const clusterOidcProviderUrl = clusterOidcProvider.url;
const clusterOidcArn = clusterOidcProvider.arn;

const autoscalerAssumeRolePolicy = pulumi.all([clusterOidcProviderUrl, clusterOidcArn]).apply(([url, arn]) => aws.iam.getPolicyDocument({
  statements: [
    {
      effect: 'Allow',
      principals: [
        {
          identifiers: [arn],
          type: 'Federated'
        }
      ],
      actions: ['sts:AssumeRoleWithWebIdentity'],
      conditions: [
        {
          test: 'StringEquals',
          values: ['system:serviceaccount:kube-system:autoscaler-aws-cluster-autoscaler'],
          variable: `${url}:sub`
        }
      ],
    }
  ]
})
);

const autoscalerRole = new aws.iam.Role('cluster-autoscaler', {
  assumeRolePolicy: autoscalerAssumeRolePolicy.json
});

const autoscalerPolicy = new aws.iam.Policy('autoscaler-policy', {
  description: pulumi.interpolate`Autoscaler policy for ${cluster.eksCluster.id}`,
  policy: JSON.stringify({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:DescribeLaunchConfigurations",
          "autoscaling:DescribeTags",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:TerminateInstanceInAutoScalingGroup",
          "ec2:DescribeLaunchTemplateVersions"
        ],
        "Resource": "*"
      }
    ]
  })
});

new aws.iam.RolePolicyAttachment('autoscaler-role-attach-policy', {
  policyArn: autoscalerPolicy.arn,
  role: autoscalerRole.name
});

const autoscaler = new k8s.helm.v3.Chart('autoscaler', {
  namespace: kubeSystemNamespace,
  chart: 'cluster-autoscaler',
  fetchOpts: {
    repo: '<https://kubernetes.github.io/autoscaler>'
  },
  version: '9.10.7',
  values: {
    cloudProvider: 'aws',
    rbac: {
      serviceAccount: {
        annotations: {
          '<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>': autoscalerRole.arn
        }
      }
    },
    awsRegion: config.get("aws.region"),
    autoDiscovery: {
      enabled: true,
      clusterName: cluster.eksCluster.name
    }
  }
}, {
  provider: cluster.provider,
  dependsOn: [cluster, metricsServerChart]
});

I hope it will prove helpful to others. And I welcome any feedback on the code, patterns or if I am doing anything wrong or wrong way.

looks great!