https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# aws
s
This message was deleted.
c
Hi, Can you open a git issue for this and add the verbose logging to it: 
pulumi up --logtostderr -v9 --debug
p
sure thing.
b
@purple-orange-91853 i knew we worked through some creds things the other day - the last time this happened to me the session token had expired. Could you refresh your creds and try again?
p
I am using creds that don't expire at the moment to remove that variable. I've narrowed it down to some type of issue with helm deployments and k8s package deployements as if I remove those from the stack the process will complete as expected
b
are you setting a provider on your Kubernetes resources, or using the 
KUBECONFIG
var?
p
I'm not entirely certain, how would I check that?
Copy code
new aws.Provider('aws-provider', {
    profile: profileName,
    region: awsRegion as pulumi.Input<aws.Region>
  })
  
const kubeconfigOpts = { profileName: profileName }
  deployKubeStateMetrics({ provider: cluster.provider, parent: rpaNodeGroup })
  new k8s.yaml.ConfigFile(
    'metrics-server',
    {
      file:
        '<https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml>'
    },
    { provider: cluster.provider }
  )
I've also tried downgrading the cli sdk to 2.25.2 and am getting the same thing. This is our package.json dependency list.
Copy code
{
  "dependencies": {
    "@pulumi/aws": "^4.1.0",
    "@pulumi/awsx": "^0.30.0",
    "@pulumi/eks": "^0.30.0",
    "@pulumi/kubernetes": "^3.0.0",
    "@pulumi/pulumi": "^3.1.0",
    "lodash": "^4.17.21"
  },
  "devDependencies": {
    "@types/node": "^15.0.1"
  }
}
b
okay so you're using the EKS cluster provider, can you export the kubeconfig and verify it works?
Copy code
export const kubeconfig = cluster.provider.kubeconfig
I think
p
I am not entirely certain where to place that?
b
at the end of your file would work, although I now realise your stack isn't updating so it may not help.. 😞 my hypothesis is that the provider is using the EKS auth and that isn't using a valid profile or creds, but i'm trying to figure out how to fix that
p
which would be very weird as I'm trying to create a new cluster
b
the eks provider creates a kubeconfig with your profile in it, it looks like this: https://gist.github.com/jaxxstorm/58f2ab461a71afdf5a0e6c34d3758f57 if the profile you've set can't auth to the cluster, it'll behave this way
p
that makes sense. I'm looking over the documentation and I'm unclear if you can assign a custom authconfigmap during creation. Is that possible?
b
not at the moment: https://github.com/pulumi/pulumi-eks/issues/568 you can update it inside an apply with the kubernetes SDK, or import it instead
p
so if I'm creating a new cluster, using an AWS user/role that has full admin, shouldn't the cluster still be created using the context of the aws profile being passed in from pulumi env. yaml file?
b
actually I was wrong, we do support the role mappings: https://www.pulumi.com/docs/reference/pkg/eks/cluster/#rolemapping
p
I did find that same thing about 5 minutes ago and tried to add that in, and still saw the same behaviour
Copy code
roleMappings: [ 
      {
        groups: ['system:bootstrappers','system:nodes','system:master'],
        roleArn: "arn:aws:iam::acct:role/role1",
        username: "role1"
      },
      {
        groups: ['system:bootstrappers','system:nodes','system:master'],
        roleArn: "arn:aws:iam::acct:role/role2",
        username: "role2"
      },
    ],
at the moment I've traced it to being an issue with two k8s service deployments. KubeStateMetrics, and the Datadog helm chart
if I comment those out the stack will return
Copy code
// deployKubeStateMetrics({ provider: cluster.provider, parent: rpaNodeGroup })
  // new k8s.yaml.ConfigFile(
  //   'metrics-server',
  //   {
  //     file:
  //       '<https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml>'
  //   },
  //   { provider: cluster.provider }
  // )
and
Copy code
// /** deploy datadog helm chart */
  // deployDatadog({
  //   deploymentName,
  //   parent: rpaNodeGroup,
  //   provider: cluster.provider
  // })
b
i think the best course of action here is to comment all the Kubernetes parts like you have done. export the kubeconfig: https://github.com/pulumi/examples/blob/master/aws-ts-eks/index.ts#L20 rerun your pulumi up so the values gets exported then check your kubeconfig actually works:
Copy code
pulumi stack output kubeconfig > /tmp/kubeconfig
export KUBECONFIG=/tmp/kubeconfig
kubectl cluster-info
kubectl get nodes
p
I ran pulumi and started creating resources and it finally failed out with this error. It seems as since it appears that the process is trying to create the cluster resources with the role and not the user account I'm authenticated with. I'm changing my aws profile to remove the role_arn and added admin rights directly to my user and will try again to see if it works or errors again.
Copy code
error: Command failed: kubectl apply -f /var/folders/z9/fpyqxxm94ks_f43rcnbx67mh0000gn/T/tmp-112470Juedctt5MJr.tmp
    Error from server (Forbidden): error when retrieving current configuration of:
    Resource: "<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Resource=clusterroles", GroupVersionKind: "<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=ClusterRole"
    Name: "aws-node", Namespace: ""
    from server for: "/var/folders/z9/fpyqxxm94ks_f43rcnbx67mh0000gn/T/tmp-112470Juedctt5MJr.tmp": <http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> "aws-node" is forbidden: User "AdminRole-TEMP" cannot get resource "clusterroles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" at the cluster scope
    Error from server (Forbidden): error when retrieving current configuration of:
    Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount"
    Name: "aws-node", Namespace: "kube-system"
and now it appears my stack is in a completely bad state. I am unable to destroy it to start over. 
pulumi destroy --logtostderr -v9 --debug 2> errors.txt
errors.txt
And trying to delete the states manually fails as well.
Copy code
tonyelliott@Tony-Elliott-MBP:aws-us-east-1-eks/ (feature/pulumi3*) $ pulumi state delete urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster::temp
 warning: This command will edit your stack's state directly. Confirm? Yes
error: This resource can't be safely deleted because the following resources depend on it:
 * "temp-eksRole"  (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$eks:index:ServiceRole::temp-eksRole)
 * "temp-eksClusterSecurityGroup" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::temp-eksClusterSecurityGroup)
 * "temp-eksClusterInternetEgressRule" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::temp-eksClusterInternetEgressRule)
 * "temp-eksCluster" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:eks/cluster:Cluster::temp-eksCluster)
 * "temp-eks-k8s"  (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$pulumi:providers:kubernetes::temp-eks-k8s)
 * "temp-oidcProvider" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:iam/openIdConnectProvider:OpenIdConnectProvider::temp-oidcProvider)
 * "temp-provider" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$pulumi:providers:kubernetes::temp-provider)
 * "temp-nodeSecurityGroup" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::temp-nodeSecurityGroup)
 * "temp-defaultEncrypted" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$<kubernetes:storage.k8s.io/v1:StorageClass::temp-defaultEncrypted>)
 * "temp-nodeAccess" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$kubernetes:core/v1:ConfigMap::temp-nodeAccess)

Delete those resources first before deleting this one.
tonyelliott@Tony-Elliott-MBP:aws-us-east-1-eks/ (feature/pulumi3*) $ pulumi state delete urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$eks:index:ServiceRole::temp-eksRole
 warning: This command will edit your stack's state directly. Confirm? Yes
error: No such resource "urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster:index:ServiceRole::temp-eksRole" exists in the current state
tonyelliott@Tony-Elliott-MBP:aws-us-east-1-eks/ (feature/pulumi3*) $
5 Views
Open in Slack
PreviousNext
Powered by