https://pulumi.com logo
#pulumi-deployments
Title
# pulumi-deployments
n

narrow-river-17495

08/01/2023, 8:51 PM
Anyone have any pointers for how to make the Pulumi Deployments OIDC auth work for talking to an EKS cluster?
I think the provider is trying to run
aws eks get-token ....
to get credentials, but I'm not positive - just know its failing with "the server has asked for the client to provide credentials"
w

witty-candle-66007

08/03/2023, 10:29 PM
Deployments OIDC is for accessing the cloud providers. The k8s provider talks directly to the k8s cluster and so you need to configure environment variables for the deployment for the default k8s provider to use or pass it in as part of the stack config and instantiate the k8s provider in your code. See https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/#setup (This all assumes the stack being deployed does not manage the k8s cluster, nor is it being managed by another Pulumi stack from which you could get the kubeconfig via stack reference.)
n

narrow-river-17495

08/03/2023, 10:34 PM
@witty-candle-66007 in this case it actually does manage the k8s cluster and is grabbing the kubeconfig from the EKS resource. The EKS-generated kubeconfig looks like it is executing
aws eks get-token
, and as far as I can tell that should be using the AWS env vars injected by Deployments, but somewhere along the way something isn't working. I'm at a little bit of an impasse as far as how to debug it though, any pointers?
it works locally with
AWS_ACCESS_KEY_ID
,
AWS_SECRET_ACCESS_KEY
, and
AWS_SESSION_TOKEN
set (and no other means of authenticating to k8s that I can imagine), and as far as I can tell Deployments is able to auth to AWS just fine
w

witty-candle-66007

08/03/2023, 10:39 PM
Let me run a quick test on my end.
n

narrow-river-17495

08/03/2023, 10:40 PM
and it seems a little suspicious to me that its not eg printing some kind of error from that
get-token
invocation
thanks, appreciate any help I can get on this
w

witty-candle-66007

08/03/2023, 10:46 PM
It might be a bit longer than “quick” if only because it takes a minute to stand up an eks cluster, but I’ll let you know what I find out.
n

narrow-river-17495

08/03/2023, 10:47 PM
heh I hear that, appreciate it
w

witty-candle-66007

08/03/2023, 11:39 PM
I’m seeing the issue in my test as well. When you mention the
EKS-generated kubeconfig
can you clarify what you mean by that? How are you getting the kubeconfig in your code?
n

narrow-river-17495

08/03/2023, 11:42 PM
Copy code
cluster, err := eks.NewCluster(ctx, "platform-eks", &eks.ClusterArgs{
		// Omitted a bunch of junk here
	})
	if err != nil {
		return nil, err
	}

	eksProvider, err := kubernetes.NewProvider(ctx, "platform-eks-provider", &kubernetes.ProviderArgs{
		Kubeconfig: cluster.KubeconfigJson,
	})
	if err != nil {
		return err
	}
(those aren't actually side by side like that but you get the idea)
w

witty-candle-66007

08/03/2023, 11:43 PM
Yeah. And is that using the pulumi/eks package?
or the aws.eks resource directly?
n

narrow-river-17495

08/03/2023, 11:44 PM
the Pulumi one,
"<http://github.com/pulumi/pulumi-eks/sdk/go/eks|github.com/pulumi/pulumi-eks/sdk/go/eks>"
w

witty-candle-66007

08/03/2023, 11:44 PM
yep - that’s what I assumed - just wanted to make sure.
Let me look into this further.
n

narrow-river-17495

08/03/2023, 11:46 PM
appreciate it, let me know if there's anything I can do to assist
w

witty-candle-66007

08/03/2023, 11:46 PM
will do
Can you confirm what versions of the aws and eks packages you are using?
n

narrow-river-17495

08/04/2023, 2:44 PM
<http://github.com/pulumi/pulumi-eks/sdk|github.com/pulumi/pulumi-eks/sdk> v1.0.2
and
<http://github.com/pulumi/pulumi-aws/sdk/v5|github.com/pulumi/pulumi-aws/sdk/v5> v5.42.0
w

witty-candle-66007

08/07/2023, 1:20 PM
With code I have that reproduces your error message, using Pulumi Deployments to deploy the stack from scratch works - so it’s not an OIDC interaction. Instead, at least in my case, it’s due to different authentication methods between my command line
pulumi up
and the OIDC used in Pulumi deployments. In a nutshell, the kubeconfig that is generated on initial deployment on my command line is not able to be used by deployments since the credentials are different. I haven’t tested it yet, but leveraging the ProviderCredentialOpts property as per this github issue comment, https://github.com/pulumi/pulumi-eks/issues/669#issuecomment-1429190235property should work.