Channels
#kubernetes
Title
f
flat-laptop-90489
05/23/2022, 8:23 PMI'm trying to build out an EKS project in go, and I'm having some issues with credentials and providers. Wondering if anyone has tried this particular scenario yet.
We have a Hashicorp Vault AWS credential backend already configured for this account. SO - I have pulumi grabbing some AWS credentials from Vault, building an AWS provider, and then using that for all of my AWS resources. This works well for most things (VPC created already, etc.)
But, when I use pulumi-eks to build an EKS cluster with this provider, it fails to validate that the cluster is running. From the error messages, it appear that the pulumi operator (which is running this code), is trying to use its own credentials to access the cluster afterwards.
SO, I think there's some issue with the Kubernetes provider that gets generated not using the AWS provider credentials that I used to create the cluster. At this point, I have no clue how to get around this, other than to not use the pulumi-eks library, and instead build out the cluster and node groups, etc. myself using the "standard" aws library. This would allow me to control the K8s provider that gets generated and the aws-auth configmap myself (I think)
Am I missing something, or is that the correct path forward here? Also, let me know if I should post this in AWS or Go instead
o
orange-policeman-59119
05/23/2022, 9:10 PMSo I make sure I understand, you're:
ā¢ Using Kubernetes Operator to run pulumi stacks in an existing cluster, using a gitops workflow
ā¢ Using Vault secrets configuration with the operator and finding this works for other deployments, that is you're finding secrets configured in
Pulumi.$stackName.yamleksOr maybe I have that 3rd/4th step wrong, is the
eksf
flat-laptop-90489
05/24/2022, 2:55 PMYep, 3 should just be
eks provider gives errornot trying to deploy yet
Looking at the option
ProviderCredentialOptsHmmm, actually, thinking through that, I have no clue how to do that.
Can I generate a kubernetes provider that uses aws credentials that I retrieve from Vault? Or do I need to create a local aws config on the fly that has profiles?
I have gotten myself into quite the auth pickle š
I'm going to rethink how I do credentials in general here. I'll report back if I come up with something.
o
orange-policeman-59119
05/24/2022, 3:45 PMWhat option(s) are you passing to the EKS cluster?
f
flat-laptop-90489
05/24/2022, 3:47 PM Copy code
cluster, err := eks.NewCluster(ctx, awsCluster.Name, &eks.ClusterArgs{
Version: pulumi.String("1.21"),
Name: pulumi.String(awsCluster.Name),
PublicSubnetIds: pulumi.ToStringArrayOutput(publicSubnetIds),
PrivateSubnetIds: pulumi.ToStringArrayOutput(privateSubnetIds),
VpcId: awsCluster.vpc.ID(),
InstanceRoles: iam.RoleArray{
awsCluster.nodeIamRole,
},
ClusterSecurityGroupTags: mergeDefaultTags(pulumi.StringMap{
"Name": pulumi.String(awsCluster.Name),
"Cluster": pulumi.String(awsCluster.Name),
}),
Tags: mergeDefaultTags(pulumi.StringMap{
"Name": pulumi.String(awsCluster.Name),
"Cluster": pulumi.String(awsCluster.Name),
}),
SkipDefaultNodeGroup: pulumi.BoolPtr(true),
RoleMappings: eks.RoleMappingArray{
eks.RoleMappingArgs{
// TODO: Probably need to create a special role for this
RoleArn: pulumi.String("arn:aws:iam::REDACTED:role/FairwindsAdministrator"),
Groups: pulumi.StringArray{pulumi.String("kubernetes-admins")},
Username: pulumi.String("arn:aws:iam::REDACTED:role/FairwindsAdministrator"),
},
},
ProviderCredentialOpts: eks.KubeconfigOptionsArgs{
RoleArn: pulumi.String("arn:aws:iam::REDACTED:role/FairwindsAdministrator"),
},
}, pulumi.Provider(awsCluster.provider))
if err != nil {
return err
}o
orange-policeman-59119
05/24/2022, 3:47 PMWhere awsCluster is a...?
f
flat-laptop-90489
05/24/2022, 3:49 PM_suddenly very worried I'm doing everything wrong š _
Copy code
type Cluster struct {
Name string `yaml:"name"`
Region string `yaml:"region"`
VpcCidr string `yaml:"vpcCidr"`
Subnets []Subnet `yaml:"subnets"`
StaticNodeGroup StaticNodeGroup `yaml:"staticNodeGroup"`
AddOns AddOns `yaml:"addOns"`
// pulumi created objects to find the outputs later
provider *aws.Provider
vpc *ec2.Vpc
igw *ec2.InternetGateway
azList []string
cluster *eks.Cluster
staticNodeGroup *eks.ManagedNodeGroup
nodeIamRole *iam.Role
}o
orange-policeman-59119
05/24/2022, 3:49 PMBtw, to use Hashi Vault secrets directly (not via code but by Pulumi engine/config) which might solve some issues:
https://gist.github.com/viveklak/05344c9c684dce4dea41bb09915903e0#file-operatorwithvaultsecretmanager-md
Oh yeah, I'm not sure what that is
f
flat-laptop-90489
05/24/2022, 3:50 PMThat Cluster struct is custom to my code
I saw that, and that's great for encrypting the secrets. But this is aws sts credentials for accessing the AWS API
o
orange-policeman-59119
05/24/2022, 3:50 PMOhh okay
f
flat-laptop-90489
05/24/2022, 3:50 PMYeah, sorry, not super clear š
o
orange-policeman-59119
05/24/2022, 3:50 PMSo you've built a provider via aws.NewProvider on that struct
f
flat-laptop-90489
05/24/2022, 3:51 PMYeah.
So I think I need the generated kubeconfig to set aws auth env variables in the kubeconfig with the actual credentials from that provider I created with aws.NewProvider. Which normally wouldn't be safe, but I'm only ever going to use that particular kubeconfig in my code, never export it
If eks.KubeConfigOptionsArgs supported arbitrary
envo
orange-policeman-59119
05/24/2022, 3:54 PMIf you look at the logs on the operator or Pulumi service console, do you see which resource or step is failing?
f
flat-laptop-90489
05/24/2022, 3:56 PMEssentially every step that tries to run a kubectl command.
kubernetes:core/v1:ConfigMapeks:index:VpcCnio
orange-policeman-59119
05/24/2022, 4:10 PMInteresting. I wonder what the issue is - EKS should be generating its own internal kubeconfig & provider to access the cluster
I think we're at the limit of my knowledge of the provider, but it'd be worth searching for that or creating an issue on the Pulumi EKS GitHub repo. @sparse-park-68967 any other advice?
f
flat-laptop-90489
05/24/2022, 4:15 PMAppreciate the help. I'm going to keep tinkering to see if I can come up with something. If I can't, I'll file an issue. This just may be an unnecessarily complicated credential config too š
I drew a picture of what I'm trying to do, not sure if it's helpful, but I needed it for my own thinking as well.
o
orange-policeman-59119
05/24/2022, 6:04 PMThat makes sense, tbh
Are you wrapping the values you get from Vault in
pulumi.Secret()s
sparse-park-68967
05/24/2022, 6:16 PMHi just catching up here
are you able to take the operator out of the equation to simplify for now? e.g. try to do it locally instead and see if the problem persists?
Ah I think you just opened https://github.com/pulumi/pulumi-eks/issues/712 - thanks for doing that. It does seem like there is a context gap in terms of truly discovering the authentication mechanism and the limited options supported in passing the authentication context to the kubernetes provider created by EKS. I will get that prioritized
š 1
f
flat-laptop-90489
05/24/2022, 7:01 PMThank you!
Friel, I am not wrapping those currently, just sticking them into an AWS provider. Should I be doing that first?
o
orange-policeman-59119
05/24/2022, 7:02 PMYeah, we recommend any external matter you want to ensure is encrypted in our state files & redacted in logs is passed through our Secret wrapper.
f
flat-laptop-90489
05/24/2022, 7:02 PMCool. Thanks for the heads up.
7 Views