pulumi-community #kubernetes

kubernetes

most-lighter-95902

12/05/2022, 4:46 PM

Hi, I’m trying to create

pvc

deployment

(sequentially) in two separate pulumi stacks in CI/CD. My

StorageClass

has binding mode of

WaitForFirstConsumer

which means the first

pvc

stack is stuck at PENDING state, which means the pipeline fails before it attempts to create the

deployment

. Is this a correct use case for

skipAwait

so that

deployment

stack will run before

pvc

stack times out?

dry-keyboard-94795

12/06/2022, 4:44 PM

hmm, in python, it looks as if

yaml.ConfigFile

doesn't respect

opts.providers

, only

opts.provider

. Adding the explicit

.provider

also results in warning spam about this issue, even though the providers are identical, ie:

opts=p.ResourceOptions(provider=k8s_provider, providers=[k8s_provider])

I've a workaround already, just flagging it. Was hoping to move to using

.providers

going forward, as we have Component Resources that utilise multiple providers

sticky-horse-12214

12/06/2022, 9:25 PM

@gorgeous-egg-16927 I'm working on implementing the changes we worked on in this PR that have now been released, but I'm realizing that I may have failed to expose the

allow_null_values

argument in the

helm.Release

object in the python SDK. Does that argument need to be added to that SDK manually or did I missing something in the build process?

busy-receptionist-43812

12/07/2022, 2:09 PM

Hello, not sure this is the right channel but im getting 404 on

Pulumi plugin install resource actions

Not finding anything useful in the docs, but api is not locating latest for this. Found an issue while back where we needed to set version, but that was fixed so it will fetch latest if omitted. Anyone had similar issue? Thx

damp-honey-93158

12/07/2022, 3:01 PM

good afternoon! I'm not sure how I should declare an array of YAML as C# code (context is a deployment into AKS), any help appreciated.

schedule:

- name: "daily-backup"

schedule: "0 0 * * *"

keep: 5

storageName: fs-pvc

It's the "- name" part that has me asking questions 🙂 For the other config I've got it worked out, e.g. for the following

volume:

persistentVolumeClaim:

accessModes: [ "ReadWriteOnce" ]

resources:

requests:

storage: 6G

I just wrote:

["volume"] = new Dictionary<string, object>

{

["persistentVolumeClaim"] = new Dictionary<string, object>

{

["accessModes"] = new InputList<string> { "ReadWriteOnce" },

["resources"] = new Dictionary<string, object>

{

["requests"] = new Dictionary<string, object>

{

["storage"] = "6G"

}

Help appreciated and thank you!

microscopic-cpu-38113

12/15/2022, 8:38 AM

hello, we have a use case in which the configmap has been set to auto naming and deleteBeforeReplace is also set to false, AND the value of the data is retrieved from pulumi.all and not sure if it is for this reason, Pulumi would think that there's a change in the data of the configmap and it always delete the old configmap and create a new one. As such, it creates a cascading effect like force replacement of the kubernetes job or deployment that rely on the configmap. Is there a way to avoid Pulumi from thinking that there's change in data when we use pulumi.all to retrieve the values?

loud-balloon-25685

12/19/2022, 2:56 AM

I tried using pulumi kubernete custom resource, but i got the following error, anyone know how i can force using 'v1' instead of 'v1beta1'?

const trailingSlashMiddleware = new k8s.apiextensions.CustomResource(`${name}-trailing-slash`, {
      apiVersion: '<http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>',
      kind: 'Middleware',
      metadata: { namespace: args.namespace },
      spec: {
        redirectRegex: {
          regex: `^.*\\${args.prefix}$`,
          replacement: `${args.prefix}/`,
          permanent: false,
        },
      }, 
    }, { provider: opts?.provider });

-----------ERROR----------
warning: apiVersion "<http://apiextensions.k8s.io/v1beta1/CustomResourceDefinition|apiextensions.k8s.io/v1beta1/CustomResourceDefinition>" was removed in Kubernetes 1.22. Use "<http://apiextensions.k8s.io/v1/CustomResourceDefinition|apiextensions.k8s.io/v1/CustomResourceDefinition>" instead.
    error: resource <http://middlewares.traefik.containo.us|middlewares.traefik.containo.us> was not successfully created by the Kubernetes API server : apiVersion "<http://apiextensions.k8s.io/v1beta1/CustomResourceDefinition|apiextensions.k8s.io/v1beta1/CustomResourceDefinition>" was removed in Kubernetes 1.22. 
Use "<http://apiextensions.k8s.io/v1/CustomResourceDefinition|apiextensions.k8s.io/v1/CustomResourceDefinition>" instead.

icy-jordan-58549

12/21/2022, 4:02 PM

error: [resource plugin kubernetes-1.0.2] downloading from : failed to download plugin: kubernetes-1.0.2: 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v1.0.2-darwin-amd64.tar.gz>

clever-painter-96148

12/22/2022, 11:47 AM

tldr: how can I tell Pulumi which provider should be used to delete a resource? Here is what my code does: 1. Create a GKE cluster 2. Generate a kubeconfig 3. Create a Kubernetes provider using that kubeconfig 4. Deploy stuff on the GKE cluster using that dynamically provisioned provider This works well! However, when a remove from my code something that was deployed using that provider, Pulumi tries to delete that resource using a default provider, that tries to read ~/.kube/config, so it fails. I am looking for a way to make Pulumi use the dynamically provisioned Kubernetes provider.

steep-winter-68060

12/22/2022, 11:18 PM

Hi all! I’m struggling to get Pulumi to ignore a secret created by a Helm chart. I’m not sure what to put in the

ignoreChanges

list? I’ve tried these without any suscess:

ignoreChanges: ['data', 'data.token', 'metadata.managedFields[*]'],

kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret|helm.sh/v3:Chart$kubernetes:core/v1:Secret> (datadog/datadog-agent-cluster-agent)
++ kubernetes:core/v1:Secret (create-replacement)
    [id=datadog/datadog-agent-cluster-agent]
    [urn=urn:pulumi:development::eks-cluster::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::datadog/datadog-agent-cluster-agent|helm.sh/v3:Chart$kubernetes:core/v1:Secret::datadog/datadog-agent-cluster-agent>]
    __fieldManager     : "pulumi-kubernetes-db5ac33a" => "pulumi-kubernetes-c6b496ec"
    metadata           : {
        managedFields    : [
            [0]: {
                fieldsV1  : {
                    f:data    : {
                        f:token: {}
                    }
                }
            }
            [1]: {
                apiVersion: "v1"
                fieldsType: "FieldsV1"
                fieldsV1  : {
                    f:data    : {
                        f:token: {}
                    }
                    f:metadata: {
                        f:labels: {
                            f:<http://app.kubernetes.io/instance|app.kubernetes.io/instance>  : {}
                            f:<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: {}
                            f:<http://app.kubernetes.io/name|app.kubernetes.io/name>      : {}
                            f:<http://app.kubernetes.io/version|app.kubernetes.io/version>   : {}
                            f:<http://helm.sh/chart|helm.sh/chart>               : {}
                        }
                    }
                    f:type    : {}
                }
                manager   : "pulumi-kubernetes-c6b496ec"
                operation : "Apply"
                time      : "2022-12-22T22:24:32Z"
            }
        ]
    }

microscopic-city-74409

12/28/2022, 5:55 PM

Hi All, New to pulumi! Is it only possible to create a k8s cluster with cloud managed providers[eks/aks/gke/etc...]? Do you offer the options to create self managed or on-perm k8s clusters. Similar to tools like kops, kubeadmn or clusterapi?

sparse-hairdresser-15357

01/05/2023, 3:46 PM

Hello. We have a Pulumi managed application deployment in Kubernetes. At some point someone added an ingress outside of Pulumi and now we need to add it to the state so it can be managed by Pulumi in the future. What is the best approach for this?

pulumi refresh

does nothing. I guess because it only looks at the resources in the state.

pulumi import

fails with an authorisation error, I think because in our setup we are confined to namespaces and cannot just enumerate all the namespaces. Maybe if there is a way to specify the namespace where Pulumi needs to look this can be resolved? The error we get is:

error: Preview failed: <http://ingresses.networking.k8s.io|ingresses.networking.k8s.io> "ingress-name" is forbidden: User "user-token" cannot get resource "ingresses" in API group "<http://networking.k8s.io|networking.k8s.io>" in the namespace "default"

dry-keyboard-94795

01/05/2023, 4:01 PM

Related to the above,

Import

doesn't seem to be documented on a lot of kubernetes resources. Have the docs not generated correctly, or is this intentional? example: Ingress

many-helicopter-89037

01/06/2023, 6:08 AM

Hey all, we're planning to use

renderYamlToDirectory

as an option to dump Yaml files into our CD system. It works great with one caveat. How does file names are generated? Currently it's generating like

apps_v1-deployment-default-nginx-008d52b7.yaml

. I wanted to add some prefix to the file names so

kubectl apply

respects order of the resources. Is it possible to do?

flat-engineer-30260

01/06/2023, 12:25 PM

Hi all, How do you correctly import kubeconfig of external kubernetes which isn't created via pulumi into pulumi deployment container? We had tried to put kubeconfig content as environment variable KUBECONFIG, it doesn't work. In the meantime,

pulumi config set --secret kubernetes:kubeconfig   --path ~/.kube/config

, something like this, it won't load the file content, and it's hard to parse the yaml to string and set with pulumi secret. The only worked way is to read it from the local file

pulumi_k8s = kubernetes.Provider("pulumi_k8s", kubeconfig=(lambda path: open(path).read())("kubeconfig"))

, but it is not secure to store kubeconfig file in github. How did you do that? Confusing...

proud-pizza-80589

01/09/2023, 6:20 PM

Question, i have a semi big stack, but applying changes is super slow, as in multiple minutes to even do preview. I want to split it up in multiple stacks, but how do i keep what already exists? I have some huge disks i do not want to recreate.

square-laptop-45713

01/10/2023, 6:53 PM

Having trouble with our current state (paid customer) - we’re using a Helm Chart to install Harbor in our EKS cluster. This chart generates several

Secret

s used for encryption and mounts those as volumes. At some point during an update, the `Secret`s are no longer in the k8s cluster but Pulumi believes they are. These missing `Secret`s are preventing pods from starting and the pods are stuck in the creation state (

ContainerCreating

CreateContainerConfigError

). I’ve attempted refreshing multiple times and Pulumi still believes these `Secret`s are there and are never updated in the state. I did find these entries in the job run logs from a job I ran to Preview stack changes a few hours ago (we’re using GH Actions):

-- kubernetes:core/v1:Secret ***/***-dev-***-jobservice delete original 
   +- kubernetes:core/v1:Secret ***/***-dev-***-jobservice replace [diff: ~data]
   ++ kubernetes:core/v1:Secret ***/***-dev-***-jobservice create replacement [diff: ~data]
      kubernetes:core/v1:Secret ***/***-dev-***-trivy  
   -- kubernetes:core/v1:Secret ***/***-dev-***-registry delete original 
   +- kubernetes:core/v1:Secret ***/***-dev-***-registry replace [diff: ~data]
   ++ kubernetes:core/v1:Secret ***/***-dev-***-registry create replacement [diff: ~data]
      kubernetes:core/v1:ConfigMap ***/***-dev-***-core  
   -- kubernetes:core/v1:Secret ***/***-dev-***-core delete original 
   +- kubernetes:core/v1:Secret ***/***-dev-***-core replace [diff: ~data]
   ++ kubernetes:core/v1:Secret ***/***-dev-***-core create replacement [diff: ~data]

the

-dev-***-trivy

is the only

Secret

that remains in the cluster

bland-pharmacist-96854

01/17/2023, 12:25 PM

Good morning. How should I add an identity provider to a new EKS cluster using pulumi eks package? I set the

create_oidc_provider

to true. This creates the idp provider in the iam but it does not associate it with the cluster

bitter-twilight-16606

01/19/2023, 12:30 PM

Hi, we are upgrading Pulumi.Kubernetes from 2.7.2 to 3.23.1 and getting lots of problems. I can't get the state in position where it won't recreate a bunch of resources. Any advice here? How does pulumi know the provider has changed, where is this actually stored, because its not in the state. Is this stored in kubernetes/aks somewhere?

quiet-laptop-13439

01/24/2023, 11:33 AM

Hi, is there any way to ignore some kubeconfig changes for a provider? As an example, we want to change proxy-url setting in kubeconfig without triggering recreate for all resources dependent on a provider

gorgeous-minister-41131

01/24/2023, 6:23 PM

Is anyone using Pulumi in an ArgoCD workflow? e.g. generating manifests using the k8s YAML provider, then either feeding them or comitting them for ArgoCD to pick up for GitOps?

eager-lifeguard-95876

01/27/2023, 9:22 AM

Hello, I suspect

value_yaml_files

isn’t working. I tried deploying a helm release and passing that argument. I can see the resource being updated but when I check the resources they had the chart’s defaults… any idea what I could be missing?

quiet-leather-94755

01/27/2023, 3:57 PM

Hello! 👋 I've noticed that some changes (eg. ConfigMaps) cause Kubernetes deployments to be replaced, which means immediately terminating all the existing pods before the new deployment is verified to be up and running in a healthy state. Is there any way around this? What I guess I'd like, is for a new ConfigMap + a new Deployment to be created, then wait until it's healthy, and then destroy the existing Deployment + ConfigMap 😅

astonishing-dress-81433

01/29/2023, 8:47 AM

hello 👋 I’m trying to use the

k8s.helm.v3.Chart

resource for a fairly simple application:

const chart = new k8s.helm.v3.Chart("daskhub", {
    version: "2023.1.0",
    chart: "daskhub",
    namespace: "dev",
    fetchOpts: {
        repo: "<https://helm.dask.org/>",
    },
  }, { providers: { kubernetes: cluster.provider }});

running

pulumi up

fails with:

Error: invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to pull chart: Get "<https://helm.dask.org/daskhub-2023.1.0.tgz>": dial tcp [2606:4700:3033::6815:2751]:443: connect: no route to host

The interesting thing is that the url above seems to be perfectly valid. Does anyone has thoughts on what is going on here? Thanks!

rhythmic-whale-48997

01/30/2023, 9:46 AM

👋 hello there I'm trying to render yaml files using k8s provider

renderYamlToDirectory

and then create this file on GitHub with

@pulumi/github

However, I need to call

pulumi up

twice. Is there a way to dump yaml files and then to read them in one go? Already tried

dependsOn

but it's not working. Sample code that I'm using for PoC

// Instantiate a Kubernetes Provider and specify the render directory.
const provider = new k8s.Provider("render-yaml", {
	renderYamlToDirectory: "./samples/rendered",
	enableServerSideApply: true
});

const s3 = new kx.Secret("credentials-s3", {
	metadata: {
		name: "credentials-s3"
	},
	stringData: {
		"access-key-id": "access",
		"secret-access-key": "secret"
	}
}, {
	provider
});

fs.readdirSync(path.resolve(__dirname, "samples/rendered/1-manifest/")).forEach(file => {
	const fileContent = fs.readFileSync(path.resolve(__dirname, `samples/rendered/1-manifest/${file}`), "utf-8")

	if (file.includes("v1-configmap")) {
		const yaml = load(fileContent) as k8s.core.v1.ConfigMap;

		new github.RepositoryFile(`files-${yaml.metadata.name}`, {
			repository: tenantsRepository.name,
			file: `./tenants/${yaml.metadata.namespace}/configmaps/${yaml.metadata.name}.yaml`,
			content: fileContent,
			branch: "master",
			overwriteOnCreate: true
		})
	}
});

prehistoric-toddler-40668

01/30/2023, 3:04 PM

Hey guys, Im deploying an EKS Cluster that after the creation it generates a kubeconfig file and uses it to create a k8s provider. after the cluster and the node group is finished and the provider is live, the cluster begins installing deployments using pulumi helm v3 release. for some reason, the deployments keep failing randomly( sometimes a couple are ok and other are not and vice versa) for this reason:

Helm release "monitoring/kube-prometheus-stack" was created, but failed to initialize completely. Use Helm CLI to investigate.: failed to become available within allocated timeout. Error: Helm Release monitoring/kube-prometheus-stack: the server could not find the requested resource

after retrying with pulumi up everything is fine and it deployes. help anyone please? 🙂

many-knife-65312

01/30/2023, 5:46 PM

hi! I'm stuck and hoping someone can help me figure out how to update the kubernetes provider in pulumi. The cluster IP address changed and whenever I try to

pulumi refresh

pulumi preview

neither command works because the provider endpoint is invalid

helpful-baker-38839

01/30/2023, 10:09 PM

Hello all. I’m applying a batch/v1:Job using

pulumi_kubernetes.yaml.ConfigFile

- is it possible to get the logs from the run of that job using Pulumi? Ideally to capture and store somewhere (S3 maybe) but I’d even settle for printing it in the CI output.

strong-microphone-65970

01/31/2023, 6:11 PM

Does anyone have a good a good example of using a postrenderer with a helm release in pulumi? Currently working to migrate our helm release deployment from kustomize to Pulumi and our postrenderer looks like this

apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
    name: example
    namespace: exampleNS
spec:
    postRenderers:
        - kustomize:
            patchesStrategicMerge:
                - apiVersion: v1
                  kind: Secret
                  metadata:
                    name: example-regsecret
                  data:
                    .dockerconfigjson:

I was trying to make something like this work in Pulumi but have not had any luck so far

polite-summer-58169

02/01/2023, 1:11 PM

I have a AKS cluster where I get the cluster.NodeResourceGroup by doing .Apply(). I then use that to create role assignments for the NodeResourceGroup. This is all well and good until I do a change on the cluster. If I do a change on the cluster the Apply() method does not return the NodeResourceGroup anymore, but only an empty string, which then breaks the role assignments later in the code. What is the correct approach to doing this?