https://pulumi.com logo
Powered by
Title
s

steep-toddler-94095

07/23/2021, 12:29 AM
I used Pulumi to create an EKS cluster in another AWS account using an assumed role. But 
pulumi up
failed when trying to create kubernetes objects afterward, as part of the same stack. How can I access the EKS cluster I just created?
b

bored-table-20691

07/23/2021, 12:32 AM
@steep-toddler-94095 did you use the 
ProfileArn
flag?
s

steep-toddler-94095

07/23/2021, 12:32 AM
I used 
roleArn
in my Pulumi stack config file
b

bored-table-20691

07/23/2021, 12:34 AM
ProviderCredentialOpts: eks.KubeconfigOptionsArgs{
			ProfileName: pulumi.Sprintf("ssa-%s", awsCfg.Require("region")),
		},
I have this in my cluster creation call
Basically if your default AWS profile is not allowed to access this other cluster (which it is unlikely to be), then you will need to tell it which AWS profile to use
s

steep-toddler-94095

07/23/2021, 12:35 AM
My default AWS profile has admin access to the other AWS account through an assumed role
I read that the iam user that created the EKS cluster has admin access to it, but in this case I am not sure how it works because my user does not exist within the aws account I created the cluster in.
b

bored-table-20691

07/23/2021, 12:36 AM
I’m not sure that’s sufficient, it would need to be in the aws-auth ConfigMap for the new cluster, either your assumed role ARN (and explicitly that role ARN) or your user ID)
https://pulumi-community.slack.com/archives/CRFURDVQB/p1622745366158900
🤔 1
I asked about this here
It’s a huge shortcoming of both EKS and also how the AWS CLI works with it
Giant pain
🤕 1
s

steep-toddler-94095

07/23/2021, 12:46 AM
just to make sure i understand correctly: it's impossible to access a kubernetes cluster with a user from another AWS account immediately after creation, even if that user is able to assume the role that created the EKS cluster
b

bored-table-20691

07/23/2021, 12:48 AM
I believe that is correct.
s

steep-toddler-94095

07/23/2021, 12:48 AM
Thanks Itay!
b

bored-table-20691

07/23/2021, 12:48 AM
I am not 100% sure
But I will say you’re likely somewhere down this painful path
Actually sorry
Let me back up
I think that you could technically access it. The issue comes from how the kubeconfig works. If you look at the kubeconfig, it will have something like this call in it: 
aws eks get-token
and that invocation is very limited, it can basically either use a profile or a role ARN
In your case, I think a role ARN is what you need
So you can specify 
RoleArn
when you create the cluster
and then the provider will have that in there.
(note this is different than what you specify in your pulumi stack config - this is different than what Pulumi will use for your normal AWS calls)
s

steep-toddler-94095

07/23/2021, 12:57 AM
thanks, so i was able to get connected to the cluster locally, though pulumi up still fails when modifying cluster resources. strange.
b

bored-table-20691

07/23/2021, 12:58 AM
Define locally
Locally your 
kubeconfig
probably looks different than the one Pulumi is generating
s

steep-toddler-94095

07/23/2021, 1:07 AM
oops, i mean the one in my 
~/.kube/config
. I manually edited the 
aws-auth
configmap to have my IAM ARN (from a different account) and it appears to be working now. I guess I don't totally understand what you mean by
So you can specify 
RoleArn
 when you create the cluster
where would this be specified? in the pulumi stack config or the eks cluster parameters?
b

bored-table-20691

07/23/2021, 1:11 AM
EKS cluster parameters
Which is crazy, I know, because the EKS cluster doesn’t care - this is more to tell Pulumi how to create the local kubeconfig for the generated provider for that cluster.
s

steep-toddler-94095

07/23/2021, 1:15 AM
can you show me you example code please? I can't find the 
roleArn
input parameter in https://www.pulumi.com/docs/reference/pkg/eks/cluster/
b

bored-table-20691

07/23/2021, 1:16 AM
It’s inside providercredentialopts
s

steep-toddler-94095

07/23/2021, 1:17 AM
ahh thanks. i will try this. thanks for your time today!!
b

bored-table-20691

07/23/2021, 1:18 AM
See the snippet i pasted originally - I used profile name but rolearn is the other option
Let me know if it works - I could totally believe I’m wrong
And also no worries - you’ve answered a bunch of my questions 😀
😄 1
s

steep-toddler-94095

07/23/2021, 1:19 AM
yeah i will let you know! might be tomorrow instead
it worked! 😛artypus:
b

bored-table-20691

07/23/2021, 1:55 AM
Huzzah
🎉 1
😛: 1
😄 1
b

billowy-army-68599

07/23/2021, 4:49 AM
@bored-table-20691 thanks for helping out here, this comes up so often we probably need to create a post on it, as you said earlier, it's confusion from the kubeconfig creation 😞
b

bored-table-20691

07/23/2021, 5:15 AM
Yeah. I am hopeful the EKS team is on a path to fix it, as they’ve said they will create an EKS level API to control what users and roles can access the cluster, so it should be configurable at that level once it’s there. Timeline is 🤷‍♀️ of course.
4 Views
#kubernetesJoin Slack
Powered by