https://pulumi.com logo
Powered by
#aws
Title
# aws
s

square-laptop-45713

01/05/2023, 8:09 PM
Ran into a major issue and I want to know where the best place to file the bug is: • Updated my dependencies of the pulumi SDK and Github Actions • We use Pulumi to manage an EKS Cluster in AWS • Ran GH Actions to refresh cloud state - this worked • GH Actions automated to run an 
update
command • Pulumi decided to 
recreate
the 
aws-auth
ConfigMap in the k8s cluster - this deleted all of the role arn to user mappings and I lost access to the cluster for all users except the role used to create the cluster • Currently (as I’m sure you’re aware) there is no way to automate the 
aws-auth
ConfigMap in EKS (per this issue) so it must be manually edited • from the diff besides the clobbering of the manually added arn role mappings, I saw that 
metaData.managedFields.0.manager
changed from 
"pulumi-resource-kubernetes"
=> 
"pulumi-kubernetes"
• not saying this 👆 is the reason for the 
recreate
operation but I don’t see anything else atm and this seems related to an upgrade of my dependencies • I can’t find anything in our IaC that manages that ConfigMap so it’s definitely something internal to Pulumi/Kubernetes that is managing it for us
here is the diff of that resource in the job that performed the 
recreate
operation (I put placeholders for the AWS Account #s):
Copy code
eks:index:Cluster$kubernetes:core/v1:ConfigMap (kettleos-eks-dev-nodeAccess)
++ kubernetes:core/v1:ConfigMap (create-replacement)
    [id=kube-system/aws-auth]
    [urn=urn:pulumi:dev::base::eks:index:Cluster$kubernetes:core/v1:ConfigMap::kettleos-eks-dev-nodeAccess]
    __inputs           : {
        data      : {
            mapRoles: "- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab'
  username: 'system:node:{{EC2PrivateDNSName}}'
  groups:
    - 'system:bootstrappers'
    - 'system:nodes'
" => "- rolearn: arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab
  username: system:node:{{EC2PrivateDNSName}}
  groups:
    - system:bootstrappers
    - system:nodes
"
        }
    }
    data               : {
        mapRoles: "- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab'
  username: 'system:node:{{EC2PrivateDNSName}}'
  groups:
    - 'system:bootstrappers'
    - 'system:nodes'
- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/OrganizationAccountAccessRole'
  username: 'developer'
  groups:
    - 'system:masters'
- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/OrganizationAccountReadOnlyRole'
  username: 'developer'
  groups:
    - 'system:masters'
" => "- rolearn: arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab
  username: system:node:{{EC2PrivateDNSName}}
  groups:
    - system:bootstrappers
    - system:nodes
"
    }
    metadata           : {
        annotations      : {
            <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: "{"apiVersion":"v1","data":{"mapRoles":"- rolearn: 'arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab'\n  username: 'system:node:{{EC2PrivateDNSName}}'\n  groups:\n    - 'system:bootstrappers'\n    - 'system:nodes'\n"},"kind":"ConfigMap","metadata":{"labels":{"<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>":"pulumi"},"name":"aws-auth","namespace":"kube-system"}}
" => "{"apiVersion":"v1","data":{"mapRoles":"- rolearn: arn:aws:iam::[AWSDevAccount]:role/kettleos-eks-dev-ng-role-4a11bab\n  username: system:node:{{EC2PrivateDNSName}}\n  groups:\n    - system:bootstrappers\n    - system:nodes\n"},"kind":"ConfigMap","metadata":{"labels":{"<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>":"pulumi"},"name":"aws-auth","namespace":"kube-system"}}
"
        }
        creationTimestamp: "2021-04-18T04:32:06Z" => "2022-12-28T06:51:37Z"
        managedFields    : [
            [0]: {
                fieldsV1  : {
                    f:data    : {
                        .         : {}
                        f:mapRoles: {}
                    }
                }
                manager   : "pulumi-resource-kubernetes" => "pulumi-kubernetes"
                time      : "2021-04-18T04:32:06Z" => "2022-12-28T06:51:37Z"
            }
            [1]: {
                apiVersion: "v1"
                fieldsType: "FieldsV1"
                fieldsV1  : {
                    f:data: {
                        f:mapRoles: {}
                    }
                }
                manager   : "kubectl-edit"
                operation : "Update"
                time      : "2021-04-18T05:11:50Z"
            }
        ]
        resourceVersion  : "190307907" => "210893469"
        uid              : "5ea6eea4-d11e-4a72-969d-19ab38f9b590" => "16f0f6ce-e58b-44c9-8a3b-c58910640b21"
    }
Diagnostics:
m

many-telephone-49025

01/05/2023, 8:37 PM
Hi, Not sure if i misread the output what is the difference in 
system:masters
in the old cm?
s

square-laptop-45713

01/05/2023, 8:39 PM
I can check but that part is 💯 managed by pulumi’s EKS module since that is the creator’s mapping that doesn’t get touched
m

many-telephone-49025

01/05/2023, 8:43 PM
hmm
s

square-laptop-45713

01/05/2023, 8:44 PM
oh no I was wrong about that part @many-telephone-49025
the 
system:masters
entries are what we have to add manually after the cluster is created in order for people to access the k8s cluster either in the Web Console or via CLI without using the creator role
as mentioned above via the GH Issue, manually editing the 
aws-auth
ConfigMap is the only way to accomplish this so a 
recreate
really blows it apart
m

many-telephone-49025

01/05/2023, 8:47 PM
But edited in Pulumi and not on the cluster? Sorry for asking maybe an obvious qustion
s

square-laptop-45713

01/05/2023, 8:48 PM
pulumi doesn’t expose it likely b/c EKS API doesn’t expose it (that’s the reason for that GH Issue) - it can’t be automated atm except through the 
eksctl
tool afaik
so the only way to edit that ConfigMap is to login and manually edit the ConfigMap resource via kubectl
m

many-telephone-49025

01/05/2023, 8:49 PM
then I would run a pulumi refresh
to update the state with the changes in the cluster so they are in sync
s

square-laptop-45713

01/05/2023, 8:49 PM
I refreshed it
afaik it was refreshed b/c I had been running refresh jobs on every change I made
m

many-telephone-49025

01/05/2023, 8:50 PM
if you run now the refresh job it should be fine?
as they are now in sync again
s

square-laptop-45713

01/05/2023, 8:51 PM
the previous update was a refresh that succeeded
m

many-telephone-49025

01/05/2023, 8:52 PM
did you update the pulumi-kubernetes provider?
s

square-laptop-45713

01/05/2023, 8:52 PM
I’m checking
m

many-telephone-49025

01/05/2023, 8:54 PM
There was a change two weeks ago -> https://github.com/pulumi/pulumi-kubernetes/pull/2271 with the manager name
s

square-laptop-45713

01/05/2023, 8:54 PM
I saw that when I was looking into it, that’s why I pointed this out
m

many-telephone-49025

01/05/2023, 8:54 PM
You are not running in SSA mode?
s

square-laptop-45713

01/05/2023, 8:55 PM
no
we’re using the OIDC IAM Role
this infrastructure is 2 years old
the manager name shouldn’t trigger a recreate - I would expect it to trigger an update op
btw the 
pulumi/kubernetes
package changed from 
^3.22.1
to 
^3.23.1
but more importantly I think was the changes to 
pulumi/awsx
package. Here are all the dep changes:
Copy code
-        "@pulumi/aws": "^5.20.0",
-        "@pulumi/awsx": "^0.40.1",
-        "@pulumi/docker": "^3.6.0",
-        "@pulumi/kubernetes": "^3.22.1",
-        "@pulumi/pulumi": "^3.46.1",
+        "@pulumi/aws": "^5.25.0",
+        "@pulumi/awsx": "^1.0.1",
+        "@pulumi/docker": "^3.6.1",
+        "@pulumi/kubernetes": "^3.23.1",
+        "@pulumi/pulumi": "^3.50.2",
m

many-telephone-49025

01/05/2023, 8:59 PM
Oh yes
AWSX went GA with version 1.0.0
Let me check the recreate on managed fields quickly to rule out this one
hmm no
no recreate when updating from 3.22.1 tro 3.23.1 on a configmap
s

square-laptop-45713

01/05/2023, 9:14 PM
thanks
m

many-telephone-49025

01/05/2023, 9:14 PM
I did an edit and a refresh before i updated
it made a update on a field but not a recreate of the whole CM
s

square-laptop-45713

01/05/2023, 9:16 PM
did you use the previous version of the package to create the initial EKS cluster?
m

many-telephone-49025

01/05/2023, 9:17 PM
I just used a local k8s to test the only the update of the kubernetes provider
s

square-laptop-45713

01/05/2023, 9:20 PM
well I have to perform this on our production cluster at some point so you’re saying that not refreshing could be the problem
m

many-telephone-49025

01/05/2023, 9:34 PM
Would love to connect in a call with you to check before running in prod. So we could more infos about the before state.
Tomorrow I am unfortunatly OOO as it's bank holiday in Germany
s

square-laptop-45713

01/05/2023, 9:39 PM
That would be fantastic. I won’t run in prod until next week
m

many-telephone-49025

01/06/2023, 8:42 AM
Awesome! Feel free to send an invite to engin@pulumi.com with a time slot that fits you! I have currently good availability from 10-12am
2 Views
Powered by