This message was deleted.

Building on top of this, it seems i'm in a chicken and egg situation here. I tried to move the creation of the IAM role and groups to when i create the cluster so i can add them to the

roleMappings

. However, the problem is that the IAM policy for

assumeRole

is relying on the arn and url of the oidc provider created for the cluster which don't exist yet since the cluster is not created. The policy is about allowing a pod to run with a specific service account:

aws.iam.getPolicyDocument(
          {
            version: "2012-10-17",
            statements: [
              {
                effect: "Allow",
                principals: [
                  {
                    type: "Federated",
                    identifiers: [arn],
                  },
                ],
                actions: ["sts:AssumeRoleWithWebIdentity"],
                conditions: [
                  {
                    test: "StringEquals",
                    variable: `${url}:sub`,
                    values: [`${serviceAccountFullName}`],
                  },
                ],
              },
            ],
          },
          { parent: this }
        );

Which is added to a role, which in turn is supposed to be added to

roleMappings

How are we supposed to handle these cases? Create the OIDC provider ourselves and do the linking manually? Raw edit the

string

for

aws-auth

configmap and hope for the best ? Any advice ? 🙂

I've just done pretty much the same as this, for fairly similar reasons. I write in Typescript and did this:

// Create a security group for the ENIs which allows communication between the nodes and the control plane
const sg = new aws.ec2.SecurityGroup("EKS", {
	description: "Group for the EKS cluster.",
	vpcId: vpcid,
	ingress: [
		{
			description: "ssh in from the ssh host",
			fromPort: 22,
			toPort: 22,
			protocol: "tcp",
			securityGroups: [ sshg.ids[0] ],
		},
		{
			description: "https in from the ssh host",
			fromPort: 443,
			toPort: 443,
			protocol: "tcp",
			securityGroups: [ sshg.ids[0] ],
		},
	],
	egress: [
		{
			description: "allow https out to anywhere",
			fromPort: 443,
			toPort: 443,
			protocol: "tcp",
			cidrBlocks: [ "0.0.0.0/0" ],
			ipv6CidrBlocks: ["::/0"],
		},
	],
	tags: {
		name: `${nam}-EKS`,
	},
});
// Create an EKS cluster
const cluster = new eks.Cluster(nam, {
	vpcId: vpcid,
	privateSubnetIds: [ privsub[0], privsub[1], privsub[2] ],
	publicSubnetIds: [ pubsub[0], pubsub[1], pubsub[2] ],
	clusterSecurityGroup: sg,
	endpointPrivateAccess: true,
	endpointPublicAccess: false,
	nodeAssociatePublicIpAddress: false,
	instanceType: "t3a.medium",
	maxSize: 10,
	minSize: 2,
	serviceRole: serviceRole,
	createOidcProvider: true,
	roleMappings: [
		{
			groups: [ "system:masters" ],
			roleArn: adminVMrole.arn,
			username: "admin",
		},
	],
});
// and make a clusterrole to give the AdminVM role permissons to admin it. 
const adminVM = new k8s.rbac.v1.ClusterRole("AdminVM", {
	metadata: {
		name: "AdminVM",
	},
	rules: [
		{
			verbs: ["*"],
			resources: ["*"],
			apiGroups: ["*"],
		},
	],
}, { provider:  cluster.provider });
// and add a ClusterRoleBinding to tie it in to the cluster 
const adminVMRB = new k8s.rbac.v1.ClusterRoleBinding("AdminVMRB", {
	metadata: {
		name: "cluster-admin-binding",
	},
	roleRef: {
		apiGroup: "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>",
		kind: "ClusterRole",
		name: "AdminVM",
	},
	subjects: [{
			kind: "User",
			name: "admin",
	}],
}, { provider:  cluster.provider });

Then to add the managed role with the AssumeRoleWebIdentity, I did this:

const oidcUrl = cluster.core.oidcProvider?.url;
const oidcArn = cluster.core.oidcProvider?.arn;

const saAssumeRolePolicy = pulumi
    .all([oidcUrl, oidcArn])
    .apply(([url, arn]) =>
        aws.iam.getPolicyDocument({
            statements: [
                {
                    actions: ['sts:AssumeRoleWithWebIdentity'],
                    conditions: [
                        {
                            test: 'StringEquals',
                            values: [`system:serviceaccount:default:efs-csi-controller-sa`],
                            variable: `${url.replace('https://', '')}:sub`,
                        },
                        {
                            test: 'StringEquals',
                            values: [`<http://sts.amazonaws.com|sts.amazonaws.com>`],
                            variable: `${url.replace('https://', '')}:aud`,
                        },
                    ],
                    effect: 'Allow',
                    principals: [{identifiers: [arn], type: 'Federated'}],
                },
            ],
        })
    );
// Creare a manaagedPolicy to give access to sts:AssumeRoleWithWebIdentity
const webIdentityPolicy = new aws.iam.Policy(`${nam}-WebIdentityPolicy`, {policy: JSON.stringify({
	Version: "2012-10-17",
	Statement: [{
		Action: ["sts:AssumeRoleWithWebIdentity"],
		//should probably limit this to Test roles 
		Resource: "*",
		Effect: "Allow",
	}],
})});
const efsRole = new aws.iam.Role(`${nam}-efsRole`, {
	assumeRolePolicy: saAssumeRolePolicy.json,
	managedPolicyArns: [
		efsPolicy.arn,
		webIdentityPolicy.arn,
	],
}, { dependsOn: [cluster]});

Which I cribbed from here: https://github.com/jaxxstorm/pulumi-examples/blob/6207179e2c4a6edbf60628edcc8a886c360f72ab/typescript/aws/eks-platform/external-dns/index.ts#L17-L36 Thanks to @billowy-army-68599 for that 😄

Thanks for the examples @polite-napkin-90098 I am not sure if this actually solves my problem. I already do, more or less, what you suggest here with my general cluster access roles. I add them at the creation of the cluster. However, I am trying to create a new identity AND allow it to login into k8s after the cluster has been created. Basically in your example I would try to give

efsRole

some read access to some resources in k8s but in order for that to happen I find myself in need to add it to the aws-auth configmap which i cannot seem to find a way to edit after cluster creation. If you have an idea, i'm all ears, or if there's a better way, please let me know 😄

I thought this might help solve the situation you mentioned in your second post where you were creating the roles ahead of time and needing to give them permissions. I haven't worked out how to get to that configmap, in a clean fashion yet.

It does indeed. Your example is basically what i was doing until i figured out i need to add the role arn to the configmap 😅

What happens if you just call https://www.pulumi.com/registry/packages/kubernetes/api-docs/core/v1/configmap/ with the name and details of the aws-auth configmap with all the arns in there you need. Even if pulumi recreates it, it should be fine, but I think it might just update the configmap in situ, no?

That's a good idea. I did consider grabbing the config map with the same resource that you linked. Basically doing a

ConfigMap.get('default/aws-auth')

and edit the yaml string. I actually didn't think if pulumi would just do the edit automatically. I`ll give it a try and report back a bit later