pulumi-community #kubernetes

bored-spoon-83710

08/27/2022, 3:13 PM

Hello, I’m trying to develop on the Pulumi Kubernetes resource provider to add two features I’d like to have in Helm Chart resource. First, I’d like to have autodiscovery of Kubernetes version and API versions. For this, I’ve modified the provider code and didn’t have to touch the SDKs. I wanted to do a release of my modified provider so that I can use it kind of transparently in my projects. I added a plugin download URL in the schema to point to the GitHub releases of my forked repository and ran make build. First issue, the newly generated sdk/go/kubernetes/helm/v3/pulumiUtilities.go file didn’t contain the right package, helm, but v3. Is there a way to override this? I then used GoReleaser to create a prerelease of my provider (after enabling prerelease publishing and removing blobs publishing in GoReleaser config). Last step, I added a replace line in the go.mod file of my test project (replace github.com/pulumi/pulumi-kubernetes/sdk/v3 v3.21.0 => github.com/yann-soubeyrand/pulumi-kubernetes/sdk/v3 v3.22.0-ys.1) and did a pulumi preview. It did not work, pulumi didn’t download my modified provider. I then force replaced the v3.21.0 tag in my repository to point to my commit and did a release again, adapting my replace line (replace github.com/pulumi/pulumi-kubernetes/sdk/v3 v3.21.0 => github.com/yann-soubeyrand/pulumi-kubernetes/sdk/v3 v3.21.0) and this time pulumi preview worked (it downloaded my plugin and the result was the expected one). I’d like to know if this is the right way to proceed (it seems not)? Second, I’d like to be able to customize the release name used during chart templating. For this, I don’t need to modify the provider since everything is already there, I just have to modify the SDKs. I’ve modified the schema and the Go templates and it seems to work as expected for Go project. But, if I understand correctly, I have to manually modify all the other SDKs, right? What about the Java SDK which doesn’t seem to have templates?

incalculable-midnight-8291

08/30/2022, 4:18 PM

Is there some way to tag

k8s.helm.v3.Release

so that pulumi skips uninstall on a stack destroy, where the cluster itself will be taken down?

victorious-dusk-75271

08/30/2022, 8:54 PM

How do you solve this issue? because of this pulumi fails

Copy code

warning: Refreshed resource is in an unhealthy state:
    * Resource 'allrites-frontend' was created but failed to initialize
    * Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

fierce-pillow-7950

08/31/2022, 12:40 AM

looking for help using server-side apply to create a deploymentpatch that adds a toleration to the podtemplatespec. does anyone have a good example of this?

incalculable-midnight-8291

08/31/2022, 10:38 AM

Hello again! Is there some way to wait until a

k8s.core.v1.Servic

type LoadBalancer has got its external ip assigned? Edit: found it!

const externalIP = service.status.loadBalancer.ingress;

It will be an array though.

ripe-shampoo-80285

08/31/2022, 9:06 PM

Has anybody here used Karpenter instead of ClusterAutoscaler(CA) for EKS cluster node scaling? Currently we use CA with an on-demand managed node group. We are thinking about add new type node instance. With CA, we will need to add additional managed node group. Looks like this is completely unnecessary with Karpenter. So, how do we migrate from CA to Karpenter with a smooth transition? Could we have both CA & Karpenter deployed? Here is my plan, 1) deploy Karpenter; 2) scale current managed node group down to 0 instance which will force all existing pod to be moved to nodes that are managed by Karpenter provisioner I assume. 3) Finally we will decommission the managed group & undeploy CA. Will the above plan work? Is there better ways to do this? Thanks in advance for any tips.

victorious-exabyte-70545

08/31/2022, 11:49 PM

Hi all, we recently rotated our AKS certificate and now can no longer access kubernetes with the pulumi k8provider. The certificate (kubeconfig) works with lens and our deployment agents in DevOps pipelines. Any ideas on what the issue is?

ripe-russia-4239

09/01/2022, 11:25 AM

Has anyone run into this error when deploying a Helm V3 release with a chart stored in an OCI registry?

Copy code

failed to pull chart: looks like "<oci://myregistry.azurecr.io/helm/mychart>" is not a valid chart repository or cannot be reached: object required

brash-gigabyte-81569

09/01/2022, 5:22 PM

did some light searching and didn’t see anything so forgive me if this was asked and answered: Can you use a helm chart from a git repository or does it have to be a chart repository?

full-boots-69133

09/05/2022, 10:35 PM

Hi all 👋, not sure if this is the right channel for this but it is k8s related so I might ask here. I am trying to troubleshoot an issue I am having with running preview and apply against our eks cluster. Upgrade is in the works but I need to be able to apply some changes that will be used in projects downstream that appear to be working. The issue is similar if not the same as this (I am not the author of the issue). I have noticed that pulumi wants a much later version of the kubernetes provider than the versions of the npm packages we have installed and I think the issue may stem from installing a later package that upgrades

client-go

>1.23.x

. I guess the question is, how does pulumi decide which resource plugin version it requires?

future-student-37410

09/06/2022, 8:32 AM

Hey everyone, I was wondering if any can help me with my little issue here: I am working with Pulumi to deploy AKS clusters on azure with Dapr integration. My next step is where I am struggling. I want to use the Dapr Pub/Sub service, which of course requires some sort of message broker. I am using redis for this. My redis deployment is working fine, I did a fast deployment by using the docker image and just throwing it in the cluster like this (in C#):

ApiVersion = "apps/v1",

Kind = "Deployment",

Metadata = new ObjectMetaArgs

Name = "redis-deployment",

Labels =

{ "app", "redis" },

},

},

Spec = new DeploymentSpecArgs

Replicas = 1,

Selector = new LabelSelectorArgs

MatchLabels =

{ "app", "redis" },

},

},

Template = new PodTemplateSpecArgs

Metadata = new ObjectMetaArgs

Labels =

{ "app", "redis" },

},

Annotations =

{ "<http://dapr.io/enabled|dapr.io/enabled>", "true" },

{ "<http://dapr.io/app-id|dapr.io/app-id>", "redis" },

{ "<http://dapr.io/app-port|dapr.io/app-port>", "6379" },

{ "<http://dapr.io/enable-api-logging|dapr.io/enable-api-logging>", "true" }

},

},

Spec = new PodSpecArgs

Containers =

new ContainerArgs

Name = "redis-message-broker",

Image = "<http://registry.hub.docker.com/library/redis:latest|registry.hub.docker.com/library/redis:latest>",

ImagePullPolicy = "Always",

},

},

},

},

},

}, new CustomResourceOptions()

Parent = this

});

How do I get the Redis secret key out of this deployment? The other problem is, that I don't know how to translate the dapr component yaml file into Pulumi resources. The Pulumi converter throws an error saying it is not able to convert. The yaml file looks like this:

Copy code

apiVersion: <http://dapr.io/v1alpha1|dapr.io/v1alpha1>
kind: Component
metadata:
  name: pubsub
  namespace: default
spec:
  type: pubsub.redis
  version: v1
  metadata:
  - name: redisHost
    value: <REPLACE WITH HOSTNAME FROM ABOVE - for Redis on Kubernetes it is redis-master.default.svc.cluster.local:6379>
  - name: redisPassword
    secretKeyRef:
      name: redis
      key: redis-password
 # uncomment below for connecting to redis cache instances over TLS (ex - Azure Redis Cache)
  # - name: enableTLS
  #   value: true

source: https://docs.dapr.io/getting-started/tutorials/configure-state-pubsub/#next-steps Did anyone of you deploy the Dapr Pub/Sub with Redis already?

victorious-dusk-75271

09/06/2022, 12:07 PM

hi, does anyone know how to get coredns to work on tainted nodes? right now its only deploying pods to untainted nodes

victorious-dusk-75271

09/06/2022, 12:20 PM

My problem is right now DNS does not work from the tainted node. i am not really sure why

victorious-dusk-75271

09/06/2022, 10:17 PM

does anyone know how to enable serverside apply with

@pulumi/eks

victorious-dusk-75271

09/07/2022, 3:48 PM

is there anyway to make pulumi to update/patch deployment pod image instead of deleting everything?

big-potato-91793

09/07/2022, 11:28 PM

iagnostics: kubernetesapps/v1Deployment (gitlab-runner): error: resource prd126/gitlab-runner-8cyi9fr7 was not successfully created by the Kubernetes API server : Could not create watcher for PersistentVolumeClaims objects associated with Deployment "gitlab-runner-8cyi9fr7": Get "https://k8s.nonprod9.us-east-1.tktm.io:443/api/v1/namespaces/prd126/persistentvolumeclaims?watch=true": unexpected EOF Any idea of what can cause this?

victorious-dusk-75271

09/09/2022, 4:27 AM

Copy code

kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release> (primary-eks-fluent-bit):
    warning: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://xxx/openapi/v2?timeout=32s>": dial tcp: lookup xxxxxxx on 172.22.160.1:53: read udp 172.22.161.178:34533->172.22.160.1:53: i/o timeout
    error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster has been deleted, you can edit the pulumi state to remove this resource

does anyone know how to remove those? removing those causes problem with dependency and take a lot of time to fix IaC

victorious-dusk-75271

09/09/2022, 5:15 AM

its gets into dependency hell

victorious-dusk-75271

09/09/2022, 5:18 AM

Copy code

1) state file contains errors: resource urn:pulumi:staging::allrites-infrastructure::custom:resource:eks$custom:resource:AlbIngressController$aws:iam/role:Role::primary-eks-aws-loadbalancer-controller-role dependency urn:pulumi:staging::allrites-infrastructure::custom:resource:eks$custom:resource:AlbIngressController$kubernetes:core/v1:Namespace::primary-eks-alb-namespace refers to missing resource
    2) importing this file could be dangerous; rerun with --force to proceed anyway

victorious-dusk-75271

09/09/2022, 5:19 AM

even i import then using --force, i cant run up

average-pilot-3793

09/12/2022, 10:08 AM

hello, is it possible to use the

coreDns

aws.eks.Addon

with fargate in a k8s. https://www.pulumi.com/registry/packages/aws/api-docs/eks/addon/#sts=Create%20a%20Addon%20Resource the documentation provides some level of information but when applying it i run into the following error. the full kubectl describe command output is in the thread. I suspect the label should be updated in the ResourceOptions somehow to

<http://eks.amazonaws.com/fargate-profile=|eks.amazonaws.com/fargate-profile=><cluster name>

Copy code

Events:
  Type     Reason            Age                       From               Message
  ----     ------            ----                      ----               -------
  Warning  FailedScheduling  2m29s (x5494 over 3d21h)  default-scheduler  0/3 nodes are available: 3 node(s) had taint {<http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>: fargate}, that the pod didn't tolerate.

most-lighter-95902

09/15/2022, 3:33 PM

Is there a way to use the

--from-file

flag in kubectl for secret creation via Pulumi?

delightful-bear-69098

09/18/2022, 2:18 PM

Is there an how to on how to create an use service account token ? It would possible a good tutorial to have using the kubeadmin ui as example.

bored-spoon-83710

09/19/2022, 4:29 PM

Hello, in my code I have a patch resource (I’ve enabled server side apply) to patch a daemon set which is created by another resource in my code (an EKS addon). For the initial creation of my stack, if I do a

pulumi up --skip-preview

it works as intended. However, if I do a

pulumi up

I get an error during preview:

Copy code

error: Preview failed: resource kube-system/ebs-csi-node was not successfully created by the Kubernetes API server : DaemonSet.apps "ebs-csi-node" is invalid: [spec.template.metadata.labels: Invalid value: map[string]string(nil): `selector` does not match template `labels`, spec.template.spec.containers: Required value]

Is it a limitation of the provider or did I miss something? Is there a workaround (other than skipping the preview)?

brash-gigabyte-81569

09/21/2022, 9:49 PM

Besides creating another kubernete provider, is there a way to set a default namespace when you are applying a bunch of yaml via yaml.NewConfigFile(…) ?

some-continent-7311

09/29/2022, 7:40 AM

After upgrading

@pulumi/pulumi

and

@pulumi/kubernetes

^3.0.0

and running

pulumi up

I get this warning:

Copy code

W0929 13:38:59.750875   65229 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
    To learn more, consult <https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke>

What should I do exactly to get rid of the warning?

damp-honey-93158

10/05/2022, 2:36 PM

I’d like to know if its possible to use “create if not exists” semantics for a k8s secret using pulumi, has anyone done that?

damp-honey-93158

10/06/2022, 7:15 AM

good morning! recently I've been getting random errors in Helm deployments, and I wondered if anyone else has started to experience this or its just me? On the off chance, here's what I'm seeing: Diagnostics: kubernetes:helm.sh/v3:Release (creditor-testcluster): error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "dapr-operator" namespace: "" from "": no matches for kind "ServiceAccount" in version "v1" ensure CRDs are installed first this is a DAPR based Helm install - and its been working perfectly for months, it started to fail two days ago. The cluster is ephemeral. any ideas appreciated - have a good one.

damp-honey-93158

10/06/2022, 7:16 AM

also; in lots of these errors I see k8s report that the namespace is empty

lively-needle-84406

10/06/2022, 2:15 PM

Morning everyone! I am having an issue with applying Istio helm charts into my EKS cluster using Pulumi. Base:

Copy code

export const istioBaseChart = new k8s.helm.v3.Release("istio-base", {
    chart: "base",
    version: istioVersion,
    namespace: istioNamespace.metadata.name,
    repositoryOpts: {
        repo: "<https://istio-release.storage.googleapis.com/charts>",
    },
    
}, {
    dependsOn: [cluster, istioNamespace]
});

Istiod:

Copy code

export const istiod = new k8s.helm.v3.Chart("istiod", {
    chart: "istiod",
    version: istioVersion,
    namespace: istioNamespace.metadata.name,
    fetchOpts: {
        repo: "<https://istio-release.storage.googleapis.com/charts>",
    },
}, {
    dependsOn: [cluster, istioNamespace, istioBaseChart]
});

The error I am receiving is:

Copy code

kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release> (istio-base):
    error: could not get server version from Kubernetes: Get "<https://B042EE83435E804CA59AE3C4ACC5C169.sk1.us-west-1.eks.amazonaws.com/version?timeout=32s>": dial tcp: lookup <http://xxxxxxxxxxxxxx.sk1.us-west-1.eks.amazonaws.com|xxxxxxxxxxxxxx.sk1.us-west-1.eks.amazonaws.com> on [xxxx:xxxx:xxx:xxxx::x]:xx: no such host

To me, this looks like the dependency on the cluster and other resources is not being honored. Any ideas how to ensure the cluster and other resources get created properly before applying these charts?