https://pulumi.com logo
Powered by
Title
p

purple-orange-91853

04/28/2021, 3:40 PM
STACK_TRACE:
    Error
        at Object.debuggablePromise (/Users/tonyelliott/repos/temp-platform/deployments/aws-us-east-1-eks/node_modules/@pulumi/pulumi/runtime/debuggable.js:69:75)
        at /Users/tonyelliott/repos/temp-platform/deployments/aws-us-east-1-eks/node_modules/@pulumi/pulumi/runtime/invoke.js:126:45
        at Generator.next (<anonymous>)
        at fulfilled (/Users/tonyelliott/repos/temp-platform/deployments/aws-us-east-1-eks/node_modules/@pulumi/pulumi/runtime/invoke.js:18:58)
        at runMicrotasks (<anonymous>)
        at processTicksAndRejections (internal/process/task_queues.js:93:5)
    unhandled rejection: CONTEXT(105): Invoking function: tok=kubernetes:yaml:decode asynchronously
c

cool-fireman-90027

04/28/2021, 5:00 PM
Hi, Can you open a git issue for this and add the verbose logging to it: 
pulumi up --logtostderr -v9 --debug
p

purple-orange-91853

04/28/2021, 5:23 PM
sure thing.
https://github.com/pulumi/pulumi/issues/6907
b

billowy-army-68599

04/28/2021, 6:04 PM
@purple-orange-91853 i knew we worked through some creds things the other day - the last time this happened to me the session token had expired. Could you refresh your creds and try again?
p

purple-orange-91853

04/28/2021, 6:05 PM
I am using creds that don't expire at the moment to remove that variable. I've narrowed it down to some type of issue with helm deployments and k8s package deployements as if I remove those from the stack the process will complete as expected
b

billowy-army-68599

04/28/2021, 6:11 PM
are you setting a provider on your Kubernetes resources, or using the 
KUBECONFIG
var?
p

purple-orange-91853

04/28/2021, 6:18 PM
I'm not entirely certain, how would I check that?
new aws.Provider('aws-provider', {
    profile: profileName,
    region: awsRegion as pulumi.Input<aws.Region>
  })
  
const kubeconfigOpts = { profileName: profileName }
  deployKubeStateMetrics({ provider: cluster.provider, parent: rpaNodeGroup })
  new k8s.yaml.ConfigFile(
    'metrics-server',
    {
      file:
        '<https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml>'
    },
    { provider: cluster.provider }
  )
I've also tried downgrading the cli sdk to 2.25.2 and am getting the same thing. This is our package.json dependency list. 
{
  "dependencies": {
    "@pulumi/aws": "^4.1.0",
    "@pulumi/awsx": "^0.30.0",
    "@pulumi/eks": "^0.30.0",
    "@pulumi/kubernetes": "^3.0.0",
    "@pulumi/pulumi": "^3.1.0",
    "lodash": "^4.17.21"
  },
  "devDependencies": {
    "@types/node": "^15.0.1"
  }
}
b

billowy-army-68599

04/28/2021, 6:30 PM
okay so you're using the EKS cluster provider, can you export the kubeconfig and verify it works? 
export const kubeconfig = cluster.provider.kubeconfig
I think
p

purple-orange-91853

04/28/2021, 6:42 PM
I am not entirely certain where to place that?
b

billowy-army-68599

04/28/2021, 6:57 PM
at the end of your file would work, although I now realise your stack isn't updating so it may not help.. 😞 my hypothesis is that the provider is using the EKS auth and that isn't using a valid profile or creds, but i'm trying to figure out how to fix that
p

purple-orange-91853

04/28/2021, 6:59 PM
which would be very weird as I'm trying to create a new cluster
b

billowy-army-68599

04/28/2021, 7:16 PM
the eks provider creates a kubeconfig with your profile in it, it looks like this: https://gist.github.com/jaxxstorm/58f2ab461a71afdf5a0e6c34d3758f57 if the profile you've set can't auth to the cluster, it'll behave this way
p

purple-orange-91853

04/28/2021, 7:49 PM
that makes sense. I'm looking over the documentation and I'm unclear if you can assign a custom authconfigmap during creation. Is that possible?
b

billowy-army-68599

04/28/2021, 8:05 PM
not at the moment: https://github.com/pulumi/pulumi-eks/issues/568 you can update it inside an apply with the kubernetes SDK, or import it instead
p

purple-orange-91853

04/28/2021, 8:09 PM
so if I'm creating a new cluster, using an AWS user/role that has full admin, shouldn't the cluster still be created using the context of the aws profile being passed in from pulumi env. yaml file?
b

billowy-army-68599

04/28/2021, 8:21 PM
actually I was wrong, we do support the role mappings: https://www.pulumi.com/docs/reference/pkg/eks/cluster/#rolemapping
p

purple-orange-91853

04/28/2021, 8:22 PM
I did find that same thing about 5 minutes ago and tried to add that in, and still saw the same behaviour
roleMappings: [ 
      {
        groups: ['system:bootstrappers','system:nodes','system:master'],
        roleArn: "arn:aws:iam::acct:role/role1",
        username: "role1"
      },
      {
        groups: ['system:bootstrappers','system:nodes','system:master'],
        roleArn: "arn:aws:iam::acct:role/role2",
        username: "role2"
      },
    ],
at the moment I've traced it to being an issue with two k8s service deployments. KubeStateMetrics, and the Datadog helm chart
if I comment those out the stack will return
// deployKubeStateMetrics({ provider: cluster.provider, parent: rpaNodeGroup })
  // new k8s.yaml.ConfigFile(
  //   'metrics-server',
  //   {
  //     file:
  //       '<https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml>'
  //   },
  //   { provider: cluster.provider }
  // )
and 
// /** deploy datadog helm chart */
  // deployDatadog({
  //   deploymentName,
  //   parent: rpaNodeGroup,
  //   provider: cluster.provider
  // })
b

billowy-army-68599

04/28/2021, 8:38 PM
i think the best course of action here is to comment all the Kubernetes parts like you have done. export the kubeconfig: https://github.com/pulumi/examples/blob/master/aws-ts-eks/index.ts#L20 rerun your pulumi up so the values gets exported then check your kubeconfig actually works: 
pulumi stack output kubeconfig > /tmp/kubeconfig
export KUBECONFIG=/tmp/kubeconfig
kubectl cluster-info
kubectl get nodes
p

purple-orange-91853

04/28/2021, 9:29 PM
I ran pulumi and started creating resources and it finally failed out with this error. It seems as since it appears that the process is trying to create the cluster resources with the role and not the user account I'm authenticated with. I'm changing my aws profile to remove the role_arn and added admin rights directly to my user and will try again to see if it works or errors again. 
error: Command failed: kubectl apply -f /var/folders/z9/fpyqxxm94ks_f43rcnbx67mh0000gn/T/tmp-112470Juedctt5MJr.tmp
    Error from server (Forbidden): error when retrieving current configuration of:
    Resource: "<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Resource=clusterroles", GroupVersionKind: "<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=ClusterRole"
    Name: "aws-node", Namespace: ""
    from server for: "/var/folders/z9/fpyqxxm94ks_f43rcnbx67mh0000gn/T/tmp-112470Juedctt5MJr.tmp": <http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> "aws-node" is forbidden: User "AdminRole-TEMP" cannot get resource "clusterroles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" at the cluster scope
    Error from server (Forbidden): error when retrieving current configuration of:
    Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount"
    Name: "aws-node", Namespace: "kube-system"
and now it appears my stack is in a completely bad state. I am unable to destroy it to start over. 
pulumi destroy --logtostderr -v9 --debug 2> errors.txt
errors.txt
And trying to delete the states manually fails as well. 
tonyelliott@Tony-Elliott-MBP:aws-us-east-1-eks/ (feature/pulumi3*) $ pulumi state delete urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster::temp
 warning: This command will edit your stack's state directly. Confirm? Yes
error: This resource can't be safely deleted because the following resources depend on it:
 * "temp-eksRole"  (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$eks:index:ServiceRole::temp-eksRole)
 * "temp-eksClusterSecurityGroup" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::temp-eksClusterSecurityGroup)
 * "temp-eksClusterInternetEgressRule" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::temp-eksClusterInternetEgressRule)
 * "temp-eksCluster" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:eks/cluster:Cluster::temp-eksCluster)
 * "temp-eks-k8s"  (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$pulumi:providers:kubernetes::temp-eks-k8s)
 * "temp-oidcProvider" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:iam/openIdConnectProvider:OpenIdConnectProvider::temp-oidcProvider)
 * "temp-provider" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$pulumi:providers:kubernetes::temp-provider)
 * "temp-nodeSecurityGroup" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::temp-nodeSecurityGroup)
 * "temp-defaultEncrypted" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$<kubernetes:storage.k8s.io/v1:StorageClass::temp-defaultEncrypted>)
 * "temp-nodeAccess" (urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$kubernetes:core/v1:ConfigMap::temp-nodeAccess)

Delete those resources first before deleting this one.
tonyelliott@Tony-Elliott-MBP:aws-us-east-1-eks/ (feature/pulumi3*) $ pulumi state delete urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster$eks:index:ServiceRole::temp-eksRole
 warning: This command will edit your stack's state directly. Confirm? Yes
error: No such resource "urn:pulumi:temp::aws-us-east-1-eks::eks:index:Cluster:index:ServiceRole::temp-eksRole" exists in the current state
tonyelliott@Tony-Elliott-MBP:aws-us-east-1-eks/ (feature/pulumi3*) $
#awsJoin Slack
Powered by