https://pulumi.com logo
Powered by
Title
f

future-refrigerator-88869

05/09/2022, 4:43 PM
Hi, is there a way / standard to edit the 
aws-auth
configmap (or any configmap for that matter) after the cluster has been created ? In my case, I have the eks cluster created with some 
roleMappings
but some workloads that get deployed might want to create a specific least-privileged user. I'd need to edit the existing 
ConfigMap
for 
aws-auth
after cluster creation but I can't figure a way to do it. Any help is appreciated 🙌
Building on top of this, it seems i'm in a chicken and egg situation here. I tried to move the creation of the IAM role and groups to when i create the cluster so i can add them to the 
roleMappings
. However, the problem is that the IAM policy for 
assumeRole
is relying on the arn and url of the oidc provider created for the cluster which don't exist yet since the cluster is not created. The policy is about allowing a pod to run with a specific service account: 
aws.iam.getPolicyDocument(
          {
            version: "2012-10-17",
            statements: [
              {
                effect: "Allow",
                principals: [
                  {
                    type: "Federated",
                    identifiers: [arn],
                  },
                ],
                actions: ["sts:AssumeRoleWithWebIdentity"],
                conditions: [
                  {
                    test: "StringEquals",
                    variable: `${url}:sub`,
                    values: [`${serviceAccountFullName}`],
                  },
                ],
              },
            ],
          },
          { parent: this }
        );
Which is added to a role, which in turn is supposed to be added to 
roleMappings
How are we supposed to handle these cases? Create the OIDC provider ourselves and do the linking manually? Raw edit the 
string
for 
aws-auth
configmap and hope for the best ? Any advice ? 🙂
p

polite-napkin-90098

05/10/2022, 6:29 PM
I've just done pretty much the same as this, for fairly similar reasons. I write in Typescript and did this: 
// Create a security group for the ENIs which allows communication between the nodes and the control plane
const sg = new aws.ec2.SecurityGroup("EKS", {
	description: "Group for the EKS cluster.",
	vpcId: vpcid,
	ingress: [
		{
			description: "ssh in from the ssh host",
			fromPort: 22,
			toPort: 22,
			protocol: "tcp",
			securityGroups: [ sshg.ids[0] ],
		},
		{
			description: "https in from the ssh host",
			fromPort: 443,
			toPort: 443,
			protocol: "tcp",
			securityGroups: [ sshg.ids[0] ],
		},
	],
	egress: [
		{
			description: "allow https out to anywhere",
			fromPort: 443,
			toPort: 443,
			protocol: "tcp",
			cidrBlocks: [ "0.0.0.0/0" ],
			ipv6CidrBlocks: ["::/0"],
		},
	],
	tags: {
		name: `${nam}-EKS`,
	},
});
// Create an EKS cluster
const cluster = new eks.Cluster(nam, {
	vpcId: vpcid,
	privateSubnetIds: [ privsub[0], privsub[1], privsub[2] ],
	publicSubnetIds: [ pubsub[0], pubsub[1], pubsub[2] ],
	clusterSecurityGroup: sg,
	endpointPrivateAccess: true,
	endpointPublicAccess: false,
	nodeAssociatePublicIpAddress: false,
	instanceType: "t3a.medium",
	maxSize: 10,
	minSize: 2,
	serviceRole: serviceRole,
	createOidcProvider: true,
	roleMappings: [
		{
			groups: [ "system:masters" ],
			roleArn: adminVMrole.arn,
			username: "admin",
		},
	],
});
// and make a clusterrole to give the AdminVM role permissons to admin it. 
const adminVM = new k8s.rbac.v1.ClusterRole("AdminVM", {
	metadata: {
		name: "AdminVM",
	},
	rules: [
		{
			verbs: ["*"],
			resources: ["*"],
			apiGroups: ["*"],
		},
	],
}, { provider:  cluster.provider });
// and add a ClusterRoleBinding to tie it in to the cluster 
const adminVMRB = new k8s.rbac.v1.ClusterRoleBinding("AdminVMRB", {
	metadata: {
		name: "cluster-admin-binding",
	},
	roleRef: {
		apiGroup: "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>",
		kind: "ClusterRole",
		name: "AdminVM",
	},
	subjects: [{
			kind: "User",
			name: "admin",
	}],
}, { provider:  cluster.provider });
Then to add the managed role with the AssumeRoleWebIdentity, I did this: 
const oidcUrl = cluster.core.oidcProvider?.url;
const oidcArn = cluster.core.oidcProvider?.arn;

const saAssumeRolePolicy = pulumi
    .all([oidcUrl, oidcArn])
    .apply(([url, arn]) =>
        aws.iam.getPolicyDocument({
            statements: [
                {
                    actions: ['sts:AssumeRoleWithWebIdentity'],
                    conditions: [
                        {
                            test: 'StringEquals',
                            values: [`system:serviceaccount:default:efs-csi-controller-sa`],
                            variable: `${url.replace('https://', '')}:sub`,
                        },
                        {
                            test: 'StringEquals',
                            values: [`<http://sts.amazonaws.com|sts.amazonaws.com>`],
                            variable: `${url.replace('https://', '')}:aud`,
                        },
                    ],
                    effect: 'Allow',
                    principals: [{identifiers: [arn], type: 'Federated'}],
                },
            ],
        })
    );
// Creare a manaagedPolicy to give access to sts:AssumeRoleWithWebIdentity
const webIdentityPolicy = new aws.iam.Policy(`${nam}-WebIdentityPolicy`, {policy: JSON.stringify({
	Version: "2012-10-17",
	Statement: [{
		Action: ["sts:AssumeRoleWithWebIdentity"],
		//should probably limit this to Test roles 
		Resource: "*",
		Effect: "Allow",
	}],
})});
const efsRole = new aws.iam.Role(`${nam}-efsRole`, {
	assumeRolePolicy: saAssumeRolePolicy.json,
	managedPolicyArns: [
		efsPolicy.arn,
		webIdentityPolicy.arn,
	],
}, { dependsOn: [cluster]});
Which I cribbed from here: https://github.com/jaxxstorm/pulumi-examples/blob/6207179e2c4a6edbf60628edcc8a886c360f72ab/typescript/aws/eks-platform/external-dns/index.ts#L17-L36 Thanks to @billowy-army-68599 for that 😄
f

future-refrigerator-88869

05/10/2022, 8:40 PM
Thanks for the examples @polite-napkin-90098 I am not sure if this actually solves my problem. I already do, more or less, what you suggest here with my general cluster access roles. I add them at the creation of the cluster. However, I am trying to create a new identity AND allow it to login into k8s after the cluster has been created. Basically in your example I would try to give 
efsRole
some read access to some resources in k8s but in order for that to happen I find myself in need to add it to the aws-auth configmap which i cannot seem to find a way to edit after cluster creation. If you have an idea, i'm all ears, or if there's a better way, please let me know 😄
p

polite-napkin-90098

05/11/2022, 1:51 PM
I thought this might help solve the situation you mentioned in your second post where you were creating the roles ahead of time and needing to give them permissions. I haven't worked out how to get to that configmap, in a clean fashion yet.
f

future-refrigerator-88869

05/11/2022, 1:53 PM
It does indeed. Your example is basically what i was doing until i figured out i need to add the role arn to the configmap 😅
p

polite-napkin-90098

05/11/2022, 2:01 PM
What happens if you just call https://www.pulumi.com/registry/packages/kubernetes/api-docs/core/v1/configmap/ with the name and details of the aws-auth configmap with all the arns in there you need. Even if pulumi recreates it, it should be fine, but I think it might just update the configmap in situ, no?
f

future-refrigerator-88869

05/11/2022, 2:05 PM
That's a good idea. I did consider grabbing the config map with the same resource that you linked. Basically doing a 
ConfigMap.get('default/aws-auth')
and edit the yaml string. I actually didn't think if pulumi would just do the edit automatically. I`ll give it a try and report back a bit later
😃 1
4 Views
#generalJoin Slack
Powered by