pulumi-community #kubernetes

hallowed-horse-57635

11/11/2022, 1:28 AM

question on authentication (users in kubeconfig) using automation api and k8 provider: trying to deploy a "deployment" (example nginx server) using the pulumi automation api and AWS EKS cluster. The config file (Portion with users:) is as listed below. The question is how does automation API get the credentials to run the aws command. We get the error "~ kubernetesapps/v1Deployment id3 refreshing (5s) warning: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: the server has asked for the client to provide credentials". i have tried adding env variable in the kubeconfig to pass secret and access key but doesn't seem to work. (the aws command cannot use the key and secret as an argument). If i set the aws configure profile explicitly in the container where api is running, it all works fine but we don't want to hardcode the creds in the container but want to pass the kubeconfig file dynamically..... there are many example using cli but cant find any thing on how automation API works for this use case. Any pointers are appreciated. users: - name: arnawseksus west 1XXXXXXX:cluster/eks-clu01a user: exec: apiVersion: client.authentication.k8s.io/v1beta1 args: - --region - us-west-1 - eks - --cluster-name - eks-clu01a command: aws

bitter-carpenter-93554

11/14/2022, 11:05 PM

bumping up the question

damp-honey-93158

11/15/2022, 10:05 AM

Hi there! Who is running parallel Release tasks when bringing up a k8s cluster? We see weird/random errors when running parallel Helm Release instances … curious if others are experiencing this?

hallowed-horse-57635

11/15/2022, 3:39 PM

question on k8 provider. I have deployed a deployment, service and an ingress component ti an aws eks cluster. Doesn't k8 require an ingress controller and i couldn't find in in the api docs for k8 provider? Are we supposed to deploy a alb for this need?

delightful-mouse-18472

11/16/2022, 12:21 PM

Hi there, I’m building a class extending

@pulumi/kubernetes/core/v1/persistentVolumeClaim

and I want to store the current storage size used, I did manage to do that. Now, I have another challenge, I want to run an update and use the current storage size, compare it with the one that’s coming up from the update and use the biggest storage size. Why? because PVC cannot scale down. The idea is to have a Pulumi resource that self-check its current values and run some logic before invoking the

super

method

polite-napkin-90098

11/16/2022, 5:22 PM

This may be a little of topic, but I think the community here must have some insight on this. I just updated a pulumi stack which deploys the https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack helm chart If you have a look at that page you'll see that major version upgrades require the update of the CRD as helm doesn't support updating CRD, they provide yaml to do it. I found out about this the hard way by having my helm chart deploy fail, which amongst other things deleted the PV and PVC that grafana use and have all the dashboards I wrote on them. :'( So what do people do about such upgrades? Do you pin all the helm chart versions and then work out a release process for each major version upgrade? Do you backup all your PVs etc. before every upgrade, 'just in case'. I'm trying to gain confidence in the Pulumi/k8s/helm ecosystem but it seems to be full of footguns. 😕

victorious-dusk-75271

11/18/2022, 3:18 AM

Copy code

kubernetes:apps/v1:DeploymentPatch (primary-eks-coredns-deployment-patch):
    warning: Refreshed resource is in an unhealthy state:
    * Resource 'coredns' was created but failed to initialize
    * [MinimumReplicasUnavailable] Deployment does not have minimum availability.
    * Minimum number of live Pods was not attained
    * [Pod kube-system/coredns-6d5cf8bdbd-5t8lj]:
    * [Pod kube-system/coredns-6d5cf8bdbd-xztm8]:

victorious-dusk-75271

11/18/2022, 3:18 AM

How do I fix this?

victorious-dusk-75271

11/18/2022, 3:18 AM

I see coredns pods are up and running

bored-baker-95734

11/21/2022, 8:03 AM

Anybody from pulumi support is available to look up at this ? https://github.com/pulumi/pulumi-eks/issues/816

straight-arm-50771

11/21/2022, 6:26 PM

Ran into similar issue to three posts up. Tried to upgrade the argo-cd Helm chart. One of my deployments worked without issue, another is throwing:

error: failed to create chart from template: chart requires kubeVersion: >=1.22.0-0 which is incompatible with Kubernetes v1.20.0

My cluster that worked is v1.23.8-gke.1900. The one that doesn't is v1.23.12-gke.100. I don't see an open issue, is there a long-term fix inbound?

proud-pizza-80589

11/23/2022, 6:35 AM

Is there a way to manage custom kubernetes objects, I’m trying to create a Route on OpenShift.

victorious-church-57397

11/23/2022, 7:23 PM

Hello team, is there a way to pass

--feature-gate

arguments through to helm releases? i cant see any options for it in the provider

sparse-spring-91820

11/24/2022, 8:20 AM

When I run pulumi update command inside circleci I receive this error:

Error: Failed to parse kubectl version JSON output

but when I run it locally it works. This happened after I update some of the circleci orbs and pulumi plugins to the newest versions:

Copy code

// circleci config
orbs:
  node: circleci/node@4.5.1
  aws-cli: circleci/aws-cli@3.1.3
  aws-ecr: circleci/aws-ecr@7.0.0
  pulumi: pulumi/pulumi@2.1.0
  shellcheck: circleci/shellcheck@2.2.4
  kubernetes: circleci/kubernetes@1.3.1


// package.json
  "dependencies": {
    "@pulumi/aws": "^5.21.1",
    "@pulumi/awsx": "^0.40.1",
    "@pulumi/eks": "^0.42.7",
    "@pulumi/kubernetes": "^3.22.1",
    "@pulumi/pulumi": "^3.47.2"
  }

When I see output from pulumi program inside circleci it looks something like this:

Copy code

Outputs:
  - cidr            : "10.0.0.0/16"
  - provider        : {
      - id        : "some-id"
      - kubeconfig: (json) {
          - apiVersion     : "v1"
          - clusters       : [
          ...

But by running it locally I have output in this format:

Copy code

cidr            : "10.0.0.0/16"
    provider        : {
        id        : "some-id"
        kubeconfig: (json) {
            apiVersion     : "v1"
            clusters       : [
                [0]: {
                    cluster: {
            ...

It looks to me that error is thrown because of different output format, but how to change it? This is my pulumi program output:

Copy code

module.exports = {
  vpcId: vpc.id,
  publicSubnetIds: pulumi.output(vpc.publicSubnetIds),
  privateSubnetIds: pulumi.output(vpc.privateSubnetIds),
  cidr: vpc.vpc.cidrBlock,
  provider: cluster.provider
};

Before updating orbs and pulumi plugins I didn’t have this issue, output was the same both locally and on circleci but I had to update plugins because I was receiving:

Kubeconfig user entry is using deprecated API version <http://client.authentication.k8s.io/v1alpha1|client.authentication.k8s.io/v1alpha1>. Run 'aws eks update-kubeconfig' to update.

error so I had to make update Does anyone know what is the issue? Thanks in advance 🙌

straight-arm-50771

11/26/2022, 11:11 PM

Issue I reported on the 21st is considerably more prominent now. I have CI setup that uses the official docker image

pulumi/pulumi:latest

and deploys a k8s cluster & apps. Consistently getting the error stemming from Helm chart deploy:

Copy code

kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release> (argo-deploy):
    error: failed to create chart from template: chart requires kubeVersion: >=1.22.0-0 which is incompatible with Kubernetes v1.20.0

This is preventing the

pulumi up

to fire at all. update- appears to be this stack overflow report, detailing how Pulumi reports kubeversion https://stackoverflow.com/questions/74291076/helm-reads-wrong-kubeversion-1-22-0-0-for-v1-23-0-as-v1-20-0?newreg=967f4794b98e4a319902a9848e3a8a2d update2- this update to ArgoCD Helm is what uncovered this issue: https://github.com/argoproj/argo-helm/commit/3d9e2f35a6e6249c27fd4ccd8129622d886ef4ea

polite-zoo-26802

11/30/2022, 11:53 PM

Hi, I am trying to test the local.command on the Kubernetes environment - Following this doc - https://www.pulumi.com/registry/packages/command/ However, getting an error. Am I doing something wrong? posting the exception here, thanks

straight-arm-50771

12/01/2022, 11:38 PM

kubernetes:yaml:ConfigFile

transformations for YAML available yet? Trying to inject a value in a the YAML to be applied, not seeing any examples on this for YAML.

billowy-horse-79629

12/04/2022, 4:27 PM

Hey all, This is not the first time i’m encountering this annoying

transport is closing

error while upgrading Helm Releases. Actually, i’m not changing anything in my pulumi stack, but the helm release doesn’t know how to keep the ServiceAccount list in order, so Pulumi is trying to upgrade it anyway, you can see here the changes that found (nothing really change, just the order of the list) - Anyway, applying this change (pulumi up) is broken and i’m getting

transport is closing

, which happened to me a while a go and the only solution that I found was to fully destroy the k8s cluster and re-create it (Which is the worst solution I could imagine to use). I don’t want to destroy this cluster, especially by the fact that it’s a new one that I created it a week ago so nothing changed there. The full log is in the thread. I’m completely lost by this error, so any help I can get would be super appreciated.

most-lighter-95902

12/05/2022, 4:46 PM

Hi, I’m trying to create

pvc

deployment

(sequentially) in two separate pulumi stacks in CI/CD. My

StorageClass

has binding mode of

WaitForFirstConsumer

which means the first

pvc

stack is stuck at PENDING state, which means the pipeline fails before it attempts to create the

deployment

. Is this a correct use case for

skipAwait

so that

deployment

stack will run before

pvc

stack times out?

dry-keyboard-94795

12/06/2022, 4:44 PM

hmm, in python, it looks as if

yaml.ConfigFile

doesn't respect

opts.providers

, only

opts.provider

. Adding the explicit

.provider

also results in warning spam about this issue, even though the providers are identical, ie:

Copy code

opts=p.ResourceOptions(provider=k8s_provider, providers=[k8s_provider])

I've a workaround already, just flagging it. Was hoping to move to using

.providers

going forward, as we have Component Resources that utilise multiple providers

sticky-horse-12214

12/06/2022, 9:25 PM

@gorgeous-egg-16927 I'm working on implementing the changes we worked on in this PR that have now been released, but I'm realizing that I may have failed to expose the

allow_null_values

argument in the

helm.Release

object in the python SDK. Does that argument need to be added to that SDK manually or did I missing something in the build process?

busy-receptionist-43812

12/07/2022, 2:09 PM

Hello, not sure this is the right channel but im getting 404 on

Copy code

Pulumi plugin install resource actions

Not finding anything useful in the docs, but api is not locating latest for this. Found an issue while back where we needed to set version, but that was fixed so it will fetch latest if omitted. Anyone had similar issue? Thx

damp-honey-93158

12/07/2022, 3:01 PM

good afternoon! I'm not sure how I should declare an array of YAML as C# code (context is a deployment into AKS), any help appreciated.

schedule:

- name: "daily-backup"

schedule: "0 0 * * *"

keep: 5

storageName: fs-pvc

It's the "- name" part that has me asking questions 🙂 For the other config I've got it worked out, e.g. for the following

volume:

persistentVolumeClaim:

accessModes: [ "ReadWriteOnce" ]

resources:

requests:

storage: 6G

I just wrote:

["volume"] = new Dictionary<string, object>

{

["persistentVolumeClaim"] = new Dictionary<string, object>

{

["accessModes"] = new InputList<string> { "ReadWriteOnce" },

["resources"] = new Dictionary<string, object>

{

["requests"] = new Dictionary<string, object>

{

["storage"] = "6G"

}

Help appreciated and thank you!

microscopic-cpu-38113

12/15/2022, 8:38 AM

hello, we have a use case in which the configmap has been set to auto naming and deleteBeforeReplace is also set to false, AND the value of the data is retrieved from pulumi.all and not sure if it is for this reason, Pulumi would think that there's a change in the data of the configmap and it always delete the old configmap and create a new one. As such, it creates a cascading effect like force replacement of the kubernetes job or deployment that rely on the configmap. Is there a way to avoid Pulumi from thinking that there's change in data when we use pulumi.all to retrieve the values?

loud-balloon-25685

12/19/2022, 2:56 AM

I tried using pulumi kubernete custom resource, but i got the following error, anyone know how i can force using 'v1' instead of 'v1beta1'?

Copy code

const trailingSlashMiddleware = new k8s.apiextensions.CustomResource(`${name}-trailing-slash`, {
      apiVersion: '<http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>',
      kind: 'Middleware',
      metadata: { namespace: args.namespace },
      spec: {
        redirectRegex: {
          regex: `^.*\\${args.prefix}$`,
          replacement: `${args.prefix}/`,
          permanent: false,
        },
      }, 
    }, { provider: opts?.provider });

-----------ERROR----------
warning: apiVersion "<http://apiextensions.k8s.io/v1beta1/CustomResourceDefinition|apiextensions.k8s.io/v1beta1/CustomResourceDefinition>" was removed in Kubernetes 1.22. Use "<http://apiextensions.k8s.io/v1/CustomResourceDefinition|apiextensions.k8s.io/v1/CustomResourceDefinition>" instead.
    error: resource <http://middlewares.traefik.containo.us|middlewares.traefik.containo.us> was not successfully created by the Kubernetes API server : apiVersion "<http://apiextensions.k8s.io/v1beta1/CustomResourceDefinition|apiextensions.k8s.io/v1beta1/CustomResourceDefinition>" was removed in Kubernetes 1.22. 
Use "<http://apiextensions.k8s.io/v1/CustomResourceDefinition|apiextensions.k8s.io/v1/CustomResourceDefinition>" instead.

icy-jordan-58549

12/21/2022, 4:02 PM

Copy code

error: [resource plugin kubernetes-1.0.2] downloading from : failed to download plugin: kubernetes-1.0.2: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v1.0.2-darwin-amd64.tar.gz>

clever-painter-96148

12/22/2022, 11:47 AM

tldr: how can I tell Pulumi which provider should be used to delete a resource? Here is what my code does: 1. Create a GKE cluster 2. Generate a kubeconfig 3. Create a Kubernetes provider using that kubeconfig 4. Deploy stuff on the GKE cluster using that dynamically provisioned provider This works well! However, when a remove from my code something that was deployed using that provider, Pulumi tries to delete that resource using a default provider, that tries to read ~/.kube/config, so it fails. I am looking for a way to make Pulumi use the dynamically provisioned Kubernetes provider.

steep-winter-68060

12/22/2022, 11:18 PM

Hi all! I’m struggling to get Pulumi to ignore a secret created by a Helm chart. I’m not sure what to put in the

ignoreChanges

list? I’ve tried these without any suscess:

ignoreChanges: ['data', 'data.token', 'metadata.managedFields[*]'],

Copy code

kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret|helm.sh/v3:Chart$kubernetes:core/v1:Secret> (datadog/datadog-agent-cluster-agent)
++ kubernetes:core/v1:Secret (create-replacement)
    [id=datadog/datadog-agent-cluster-agent]
    [urn=urn:pulumi:development::eks-cluster::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::datadog/datadog-agent-cluster-agent|helm.sh/v3:Chart$kubernetes:core/v1:Secret::datadog/datadog-agent-cluster-agent>]
    __fieldManager     : "pulumi-kubernetes-db5ac33a" => "pulumi-kubernetes-c6b496ec"
    metadata           : {
        managedFields    : [
            [0]: {
                fieldsV1  : {
                    f:data    : {
                        f:token: {}
                    }
                }
            }
            [1]: {
                apiVersion: "v1"
                fieldsType: "FieldsV1"
                fieldsV1  : {
                    f:data    : {
                        f:token: {}
                    }
                    f:metadata: {
                        f:labels: {
                            f:<http://app.kubernetes.io/instance|app.kubernetes.io/instance>  : {}
                            f:<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: {}
                            f:<http://app.kubernetes.io/name|app.kubernetes.io/name>      : {}
                            f:<http://app.kubernetes.io/version|app.kubernetes.io/version>   : {}
                            f:<http://helm.sh/chart|helm.sh/chart>               : {}
                        }
                    }
                    f:type    : {}
                }
                manager   : "pulumi-kubernetes-c6b496ec"
                operation : "Apply"
                time      : "2022-12-22T22:24:32Z"
            }
        ]
    }

microscopic-city-74409

12/28/2022, 5:55 PM

Hi All, New to pulumi! Is it only possible to create a k8s cluster with cloud managed providers[eks/aks/gke/etc...]? Do you offer the options to create self managed or on-perm k8s clusters. Similar to tools like kops, kubeadmn or clusterapi?

sparse-hairdresser-15357

01/05/2023, 3:46 PM

Hello. We have a Pulumi managed application deployment in Kubernetes. At some point someone added an ingress outside of Pulumi and now we need to add it to the state so it can be managed by Pulumi in the future. What is the best approach for this?

pulumi refresh

does nothing. I guess because it only looks at the resources in the state.

pulumi import

fails with an authorisation error, I think because in our setup we are confined to namespaces and cannot just enumerate all the namespaces. Maybe if there is a way to specify the namespace where Pulumi needs to look this can be resolved? The error we get is:

Copy code

error: Preview failed: <http://ingresses.networking.k8s.io|ingresses.networking.k8s.io> "ingress-name" is forbidden: User "user-token" cannot get resource "ingresses" in API group "<http://networking.k8s.io|networking.k8s.io>" in the namespace "default"