f

    future-refrigerator-88869

    4 months ago
    Hi, is there a way / standard to edit the
    aws-auth
    configmap (or any configmap for that matter) after the cluster has been created ? In my case, I have the eks cluster created with some
    roleMappings
    but some workloads that get deployed might want to create a specific least-privileged user. I'd need to edit the existing
    ConfigMap
    for
    aws-auth
    after cluster creation but I can't figure a way to do it. Any help is appreciated 🙌
    Building on top of this, it seems i'm in a chicken and egg situation here. I tried to move the creation of the IAM role and groups to when i create the cluster so i can add them to the
    roleMappings
    . However, the problem is that the IAM policy for
    assumeRole
    is relying on the arn and url of the oidc provider created for the cluster which don't exist yet since the cluster is not created. The policy is about allowing a pod to run with a specific service account:
    aws.iam.getPolicyDocument(
              {
                version: "2012-10-17",
                statements: [
                  {
                    effect: "Allow",
                    principals: [
                      {
                        type: "Federated",
                        identifiers: [arn],
                      },
                    ],
                    actions: ["sts:AssumeRoleWithWebIdentity"],
                    conditions: [
                      {
                        test: "StringEquals",
                        variable: `${url}:sub`,
                        values: [`${serviceAccountFullName}`],
                      },
                    ],
                  },
                ],
              },
              { parent: this }
            );
    Which is added to a role, which in turn is supposed to be added to
    roleMappings
    How are we supposed to handle these cases? Create the OIDC provider ourselves and do the linking manually? Raw edit the
    string
    for
    aws-auth
    configmap and hope for the best ? Any advice ? 🙂
    p

    polite-napkin-90098

    4 months ago
    I've just done pretty much the same as this, for fairly similar reasons. I write in Typescript and did this:
    // Create a security group for the ENIs which allows communication between the nodes and the control plane
    const sg = new aws.ec2.SecurityGroup("EKS", {
    	description: "Group for the EKS cluster.",
    	vpcId: vpcid,
    	ingress: [
    		{
    			description: "ssh in from the ssh host",
    			fromPort: 22,
    			toPort: 22,
    			protocol: "tcp",
    			securityGroups: [ sshg.ids[0] ],
    		},
    		{
    			description: "https in from the ssh host",
    			fromPort: 443,
    			toPort: 443,
    			protocol: "tcp",
    			securityGroups: [ sshg.ids[0] ],
    		},
    	],
    	egress: [
    		{
    			description: "allow https out to anywhere",
    			fromPort: 443,
    			toPort: 443,
    			protocol: "tcp",
    			cidrBlocks: [ "0.0.0.0/0" ],
    			ipv6CidrBlocks: ["::/0"],
    		},
    	],
    	tags: {
    		name: `${nam}-EKS`,
    	},
    });
    // Create an EKS cluster
    const cluster = new eks.Cluster(nam, {
    	vpcId: vpcid,
    	privateSubnetIds: [ privsub[0], privsub[1], privsub[2] ],
    	publicSubnetIds: [ pubsub[0], pubsub[1], pubsub[2] ],
    	clusterSecurityGroup: sg,
    	endpointPrivateAccess: true,
    	endpointPublicAccess: false,
    	nodeAssociatePublicIpAddress: false,
    	instanceType: "t3a.medium",
    	maxSize: 10,
    	minSize: 2,
    	serviceRole: serviceRole,
    	createOidcProvider: true,
    	roleMappings: [
    		{
    			groups: [ "system:masters" ],
    			roleArn: adminVMrole.arn,
    			username: "admin",
    		},
    	],
    });
    // and make a clusterrole to give the AdminVM role permissons to admin it. 
    const adminVM = new k8s.rbac.v1.ClusterRole("AdminVM", {
    	metadata: {
    		name: "AdminVM",
    	},
    	rules: [
    		{
    			verbs: ["*"],
    			resources: ["*"],
    			apiGroups: ["*"],
    		},
    	],
    }, { provider:  cluster.provider });
    // and add a ClusterRoleBinding to tie it in to the cluster 
    const adminVMRB = new k8s.rbac.v1.ClusterRoleBinding("AdminVMRB", {
    	metadata: {
    		name: "cluster-admin-binding",
    	},
    	roleRef: {
    		apiGroup: "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>",
    		kind: "ClusterRole",
    		name: "AdminVM",
    	},
    	subjects: [{
    			kind: "User",
    			name: "admin",
    	}],
    }, { provider:  cluster.provider });
    Then to add the managed role with the AssumeRoleWebIdentity, I did this:
    const oidcUrl = cluster.core.oidcProvider?.url;
    const oidcArn = cluster.core.oidcProvider?.arn;
    
    const saAssumeRolePolicy = pulumi
        .all([oidcUrl, oidcArn])
        .apply(([url, arn]) =>
            aws.iam.getPolicyDocument({
                statements: [
                    {
                        actions: ['sts:AssumeRoleWithWebIdentity'],
                        conditions: [
                            {
                                test: 'StringEquals',
                                values: [`system:serviceaccount:default:efs-csi-controller-sa`],
                                variable: `${url.replace('https://', '')}:sub`,
                            },
                            {
                                test: 'StringEquals',
                                values: [`<http://sts.amazonaws.com|sts.amazonaws.com>`],
                                variable: `${url.replace('https://', '')}:aud`,
                            },
                        ],
                        effect: 'Allow',
                        principals: [{identifiers: [arn], type: 'Federated'}],
                    },
                ],
            })
        );
    // Creare a manaagedPolicy to give access to sts:AssumeRoleWithWebIdentity
    const webIdentityPolicy = new aws.iam.Policy(`${nam}-WebIdentityPolicy`, {policy: JSON.stringify({
    	Version: "2012-10-17",
    	Statement: [{
    		Action: ["sts:AssumeRoleWithWebIdentity"],
    		//should probably limit this to Test roles 
    		Resource: "*",
    		Effect: "Allow",
    	}],
    })});
    const efsRole = new aws.iam.Role(`${nam}-efsRole`, {
    	assumeRolePolicy: saAssumeRolePolicy.json,
    	managedPolicyArns: [
    		efsPolicy.arn,
    		webIdentityPolicy.arn,
    	],
    }, { dependsOn: [cluster]});
    Which I cribbed from here:https://github.com/jaxxstorm/pulumi-examples/blob/6207179e2c4a6edbf60628edcc8a886c360f72ab/typescript/aws/eks-platform/external-dns/index.ts#L17-L36 Thanks to @billowy-army-68599 for that 😄
    f

    future-refrigerator-88869

    4 months ago
    Thanks for the examples @polite-napkin-90098 I am not sure if this actually solves my problem. I already do, more or less, what you suggest here with my general cluster access roles. I add them at the creation of the cluster. However, I am trying to create a new identity AND allow it to login into k8s after the cluster has been created. Basically in your example I would try to give
    efsRole
    some read access to some resources in k8s but in order for that to happen I find myself in need to add it to the aws-auth configmap which i cannot seem to find a way to edit after cluster creation. If you have an idea, i'm all ears, or if there's a better way, please let me know 😄
    p

    polite-napkin-90098

    4 months ago
    I thought this might help solve the situation you mentioned in your second post where you were creating the roles ahead of time and needing to give them permissions. I haven't worked out how to get to that configmap, in a clean fashion yet.
    f

    future-refrigerator-88869

    4 months ago
    It does indeed. Your example is basically what i was doing until i figured out i need to add the role arn to the configmap 😅
    p

    polite-napkin-90098

    4 months ago
    What happens if you just call https://www.pulumi.com/registry/packages/kubernetes/api-docs/core/v1/configmap/ with the name and details of the aws-auth configmap with all the arns in there you need. Even if pulumi recreates it, it should be fine, but I think it might just update the configmap in situ, no?
    f

    future-refrigerator-88869

    4 months ago
    That's a good idea. I did consider grabbing the config map with the same resource that you linked. Basically doing a
    ConfigMap.get('default/aws-auth')
    and edit the yaml string. I actually didn't think if pulumi would just do the edit automatically. I`ll give it a try and report back a bit later