square-laptop-45713
01/05/2023, 8:09 PMupdate
command
• Pulumi decided to recreate
the aws-auth
ConfigMap in the k8s cluster - this deleted all of the role arn to user mappings and I lost access to the cluster for all users except the role used to create the cluster
• Currently (as I’m sure you’re aware) there is no way to automate the aws-auth
ConfigMap in EKS (per this issue) so it must be manually edited
• from the diff besides the clobbering of the manually added arn role mappings, I saw that metaData.managedFields.0.manager
changed from "pulumi-resource-kubernetes"
=> "pulumi-kubernetes"
• not saying this 👆 is the reason for the recreate
operation but I don’t see anything else atm and this seems related to an upgrade of my dependencies
• I can’t find anything in our IaC that manages that ConfigMap so it’s definitely something internal to Pulumi/Kubernetes that is managing it for usrecreate
operation (I put placeholders for the AWS Account #s):
eks:index:Cluster$kubernetes:core/v1:ConfigMap (kettleos-eks-dev-nodeAccess)
++ kubernetes:core/v1:ConfigMap (create-replacement)
[id=kube-system/aws-auth]
[urn=urn:pulumi:dev::base::eks:index:Cluster$kubernetes:core/v1:ConfigMap::kettleos-eks-dev-nodeAccess]
__inputs : {
data : {
mapRoles: "- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab'
username: 'system:node:{{EC2PrivateDNSName}}'
groups:
- 'system:bootstrappers'
- 'system:nodes'
" => "- rolearn: arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
"
}
}
data : {
mapRoles: "- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab'
username: 'system:node:{{EC2PrivateDNSName}}'
groups:
- 'system:bootstrappers'
- 'system:nodes'
- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/OrganizationAccountAccessRole'
username: 'developer'
groups:
- 'system:masters'
- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/OrganizationAccountReadOnlyRole'
username: 'developer'
groups:
- 'system:masters'
" => "- rolearn: arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
"
}
metadata : {
annotations : {
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: "{"apiVersion":"v1","data":{"mapRoles":"- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab'\n username: 'system:node:{{EC2PrivateDNSName}}'\n groups:\n - 'system:bootstrappers'\n - 'system:nodes'\n"},"kind":"ConfigMap","metadata":{"labels":{"<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>":"pulumi"},"name":"aws-auth","namespace":"kube-system"}}
" => "{"apiVersion":"v1","data":{"mapRoles":"- rolearn: arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab\n username: system:node:{{EC2PrivateDNSName}}\n groups:\n - system:bootstrappers\n - system:nodes\n"},"kind":"ConfigMap","metadata":{"labels":{"<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>":"pulumi"},"name":"aws-auth","namespace":"kube-system"}}
"
}
creationTimestamp: "2021-04-18T04:32:06Z" => "2022-12-28T06:51:37Z"
managedFields : [
[0]: {
fieldsV1 : {
f:data : {
. : {}
f:mapRoles: {}
}
}
manager : "pulumi-resource-kubernetes" => "pulumi-kubernetes"
time : "2021-04-18T04:32:06Z" => "2022-12-28T06:51:37Z"
}
[1]: {
apiVersion: "v1"
fieldsType: "FieldsV1"
fieldsV1 : {
f:data: {
f:mapRoles: {}
}
}
manager : "kubectl-edit"
operation : "Update"
time : "2021-04-18T05:11:50Z"
}
]
resourceVersion : "190307907" => "210893469"
uid : "5ea6eea4-d11e-4a72-969d-19ab38f9b590" => "16f0f6ce-e58b-44c9-8a3b-c58910640b21"
}
Diagnostics:
many-telephone-49025
01/05/2023, 8:37 PMsystem:masters
in the old cm?square-laptop-45713
01/05/2023, 8:39 PMmany-telephone-49025
01/05/2023, 8:43 PMsquare-laptop-45713
01/05/2023, 8:44 PMsystem:masters
entries are what we have to add manually after the cluster is created in order for people to access the k8s cluster either in the Web Console or via CLI without using the creator roleaws-auth
ConfigMap is the only way to accomplish this so a recreate
really blows it apartmany-telephone-49025
01/05/2023, 8:47 PMsquare-laptop-45713
01/05/2023, 8:48 PMeksctl
tool afaikmany-telephone-49025
01/05/2023, 8:49 PMsquare-laptop-45713
01/05/2023, 8:49 PMmany-telephone-49025
01/05/2023, 8:50 PMsquare-laptop-45713
01/05/2023, 8:51 PMmany-telephone-49025
01/05/2023, 8:52 PMsquare-laptop-45713
01/05/2023, 8:52 PMmany-telephone-49025
01/05/2023, 8:54 PMsquare-laptop-45713
01/05/2023, 8:54 PMmany-telephone-49025
01/05/2023, 8:54 PMsquare-laptop-45713
01/05/2023, 8:55 PMpulumi/kubernetes
package changed from ^3.22.1
to ^3.23.1
but more importantly I think was the changes to pulumi/awsx
package. Here are all the dep changes:
- "@pulumi/aws": "^5.20.0",
- "@pulumi/awsx": "^0.40.1",
- "@pulumi/docker": "^3.6.0",
- "@pulumi/kubernetes": "^3.22.1",
- "@pulumi/pulumi": "^3.46.1",
+ "@pulumi/aws": "^5.25.0",
+ "@pulumi/awsx": "^1.0.1",
+ "@pulumi/docker": "^3.6.1",
+ "@pulumi/kubernetes": "^3.23.1",
+ "@pulumi/pulumi": "^3.50.2",
many-telephone-49025
01/05/2023, 8:59 PMsquare-laptop-45713
01/05/2023, 9:14 PMmany-telephone-49025
01/05/2023, 9:14 PMsquare-laptop-45713
01/05/2023, 9:16 PMmany-telephone-49025
01/05/2023, 9:17 PMsquare-laptop-45713
01/05/2023, 9:20 PMmany-telephone-49025
01/05/2023, 9:34 PMsquare-laptop-45713
01/05/2023, 9:39 PMmany-telephone-49025
01/06/2023, 8:42 AM