https://pulumi.com logo
s

square-energy-17741

09/30/2021, 10:14 PM
Moving it from #general to #kubernetes Does anyone know an example of installing and configuring kubernetes "cluster-autoscaler" to use ManagedNodeGroup with pulumi/eks and pulumi/kubernetesx?
b

brave-ambulance-98491

09/30/2021, 10:20 PM
I've done this with
pulumi/kubernetes
, not
kubernetesx
, and I don't have a ready example. Are you looking for code, or do you just have questions?
w

worried-city-86458

09/30/2021, 10:21 PM
This is on my test branch so may disappear at some stage, and I'm not using pulumi/eks or pulumi/kubernetesx, but this may still be of interest: ā€¢ EksStack.cs#L206 ā€¢ K8sStack.cs#L567 (helm charts) ā€¢ K8sStack.cs#L554 (helm releases) Ideally I'd patch the node group asg tags for cluster autoscaler using a dynamic provider, but that's still not supported in dotnet/c#.
šŸŽ‰ 1
b

billowy-army-68599

09/30/2021, 11:31 PM
@worried-city-86458 you can probably use the
aws.ec2.Tag
resource to do this, similar mechanism to this: https://github.com/jaxxstorm/pulumi-examples/blob/main/typescript/aws/eks_subnet_tags/index.ts#L51-L57
s

square-energy-17741

09/30/2021, 11:33 PM
@worried-city-86458 is my understanding here correct - 1. Set 2 tags on the asg group -
Copy code
"<http://k8s.io/cluster-autoscaler/node-template/label/role|k8s.io/cluster-autoscaler/node-template/label/role>" - "<nodeGroup.Name>"
"<http://k8s.io/cluster-autoscaler/node-template/taint/role|k8s.io/cluster-autoscaler/node-template/taint/role>" - "NoSchedule"
2. Create a new IAM role for autoscaler using
ClusterAutoscalerPolicy.json
3. Deploy
cluster-autoscaler
helm chart with appropriate settings and passing the RoleArn to RBAC setting on the chart values.
w

worried-city-86458

09/30/2021, 11:42 PM
@billowy-army-68599 Ah yes, I'm not sure why I didn't use that instead, I'll have to take another look. Also fwiw, you don't need to tag your subnets for the eks cluster any more.
b

billowy-army-68599

09/30/2021, 11:42 PM
yeah I wrote that yonks ago and just kept it around mainly in case I needed something like it in future šŸ˜„
w

worried-city-86458

09/30/2021, 11:44 PM
You do need to tag subnets for aws load balancer controller though so that example will be useful to others using pulumi/eks
šŸ‘ 1
@square-energy-17741 yes, the policy is for irsa (iam roles for service accounts); i.e. least privilege access to aws
šŸ™ 1
s

square-energy-17741

09/30/2021, 11:52 PM
So are the only two steps is create a role with irsa policy and deploy the chart?
w

worried-city-86458

10/01/2021, 1:34 AM
It depends. Read https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup carefully.
<http://k8s.io/cluster-autoscaler/enabled|k8s.io/cluster-autoscaler/enabled>
tag is applied automatically by managed node group iirc However, if you're applying labels or taints to the node group (like I am here), then you should also apply hints via asg tags, i.e. step 1.
This works around https://github.com/aws/containers-roadmap/issues/608 until managed node groups eventually automatically apply tags for labels and taints
@billowy-army-68599 changed to use `Aws.AutoScaling.Tag`:
Copy code
// node group asg tags for cluster autoscaler; workaround <https://github.com/aws/containers-roadmap/issues/608>
Logger.LogDebug("Creating node group asg tags");
managedNodeGroup.Resources.Apply(resources =>
{
    var asgNames = resources.SelectMany(resource => resource.AutoscalingGroups).Select(asg => asg.Name!).ToArray();
    foreach (var asgName in asgNames)
    {
        new Tag($"{awsEksPrefix}-nodes-{nodeGroup.Name}-label",
            new TagArgs
            {
                AutoscalingGroupName = asgName,
                TagDetails = new TagTagArgs
                {
                    Key = "<http://k8s.io/cluster-autoscaler/node-template/label/role|k8s.io/cluster-autoscaler/node-template/label/role>",
                    Value = nodeGroup.Name,
                    PropagateAtLaunch = true
                }
            },
            new CustomResourceOptions { Provider = awsProvider });

        new Tag($"{awsEksPrefix}-nodes-{nodeGroup.Name}-taint",
            new TagArgs
            {
                AutoscalingGroupName = asgName,
                TagDetails = new TagTagArgs
                {
                    Key = "<http://k8s.io/cluster-autoscaler/node-template/taint/role|k8s.io/cluster-autoscaler/node-template/taint/role>",
                    Value = "NoSchedule",
                    PropagateAtLaunch = true
                }
            },
            new CustomResourceOptions { Provider = awsProvider });
    }
    return resources;
});
I think the reason I avoided this was because it has to create the tags inside apply, but this is still much better that using the aws sdk directly.
b

billowy-army-68599

10/01/2021, 2:33 AM
looks great! sometimes inside an Apply isn't the worst option
šŸ‘ 1
s

square-energy-17741

10/08/2021, 3:42 PM
@worried-city-86458 @billowy-army-68599 Thank you for your help. I have been able to put together first working cut of code to setup autocluster on EKS using managedNodeGroup in Typescript -
Copy code
// Setup Kubernetes Autoscaler

const clusterOidcProvider = cluster.core.oidcProvider;
const clusterOidcProviderUrl = clusterOidcProvider.url;
const clusterOidcArn = clusterOidcProvider.arn;

const autoscalerAssumeRolePolicy = pulumi.all([clusterOidcProviderUrl, clusterOidcArn]).apply(([url, arn]) => aws.iam.getPolicyDocument({
  statements: [
    {
      effect: 'Allow',
      principals: [
        {
          identifiers: [arn],
          type: 'Federated'
        }
      ],
      actions: ['sts:AssumeRoleWithWebIdentity'],
      conditions: [
        {
          test: 'StringEquals',
          values: ['system:serviceaccount:kube-system:autoscaler-aws-cluster-autoscaler'],
          variable: `${url}:sub`
        }
      ],
    }
  ]
})
);

const autoscalerRole = new aws.iam.Role('cluster-autoscaler', {
  assumeRolePolicy: autoscalerAssumeRolePolicy.json
});

const autoscalerPolicy = new aws.iam.Policy('autoscaler-policy', {
  description: pulumi.interpolate`Autoscaler policy for ${cluster.eksCluster.id}`,
  policy: JSON.stringify({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "autoscaling:DescribeAutoScalingGroups",
          "autoscaling:DescribeAutoScalingInstances",
          "autoscaling:DescribeLaunchConfigurations",
          "autoscaling:DescribeTags",
          "autoscaling:SetDesiredCapacity",
          "autoscaling:TerminateInstanceInAutoScalingGroup",
          "ec2:DescribeLaunchTemplateVersions"
        ],
        "Resource": "*"
      }
    ]
  })
});

new aws.iam.RolePolicyAttachment('autoscaler-role-attach-policy', {
  policyArn: autoscalerPolicy.arn,
  role: autoscalerRole.name
});

const autoscaler = new k8s.helm.v3.Chart('autoscaler', {
  namespace: kubeSystemNamespace,
  chart: 'cluster-autoscaler',
  fetchOpts: {
    repo: '<https://kubernetes.github.io/autoscaler>'
  },
  version: '9.10.7',
  values: {
    cloudProvider: 'aws',
    rbac: {
      serviceAccount: {
        annotations: {
          '<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>': autoscalerRole.arn
        }
      }
    },
    awsRegion: config.get("aws.region"),
    autoDiscovery: {
      enabled: true,
      clusterName: cluster.eksCluster.name
    }
  }
}, {
  provider: cluster.provider,
  dependsOn: [cluster, metricsServerChart]
});
I hope it will prove helpful to others. And I welcome any feedback on the code, patterns or if I am doing anything wrong or wrong way.
b

billowy-army-68599

10/08/2021, 3:43 PM
looks great!
šŸ™ 1
4 Views