Super weird, destroying a cluster with AWS elb CDR’s, destroyed the nodepool, but left the CRD instances (not deployed via this stack) deadlocking the deploy completely. The CRD would not delete as stuff was still using it so the destroy fails. Apart from destroying in the right order, any way to prevent this deadlocking?

FYI - https://pulumi-community.slack.com/archives/CB36DSVSA/p1631645316033200

Cross-posting from #general, since I just realized this channel exists.

Hey, Im trying to use aggregated api layer from pulumi, e.g. this here: https://github.com/gardener/gardener/blob/master/example/90-shoot.yaml Unfortunately there is no crd definition for it to use with

crd2pulumi

. I have been able to make it work by writing all fields manually into something like this:

new k8s.apiextensions.CustomResource(
         "shoot",
             {
             apiVersion: "core.gardener.cloud/v1beta1",
             kind: "Shoot",
...

but its kinda hacky… Does anyone have an idea how to generate a typeful and documented

pulumi.CustomResource

class for something like this? The api is actually pretty well defined: https://github.com/gardener/gardener/blob/master/pkg/apis/core/types_shoot.go

solved: i'm hoping to trying out the new

Helm Release

but with

@pulumi/kubernetes-v3.7.1

I get

TSError: ⨯ Unable to compile TypeScript:
    index.ts(67,33): error TS2339: Property 'v3' does not exist on type 'typeof import("/home/user/Projects/pulumi/k8s/node_modules/@pulumi/kubernetes/core/index")'.

in messing with the new helm.release functionality i would like to know if there is a way to get the rendered yaml (before it is deployed - like a noop run) to debug issues like :-

Previewing update (ipd-eks-use1-sfx):
     Type                              Name                          Plan       Info
 +   pulumi:pulumi:Stack               pulumi-test-ipd-eks-use1-sfx  create     1 message
 +   ├─ pulumi:providers:kubernetes    k8s_provider                  create
     └─ kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release>  signalfx-agent                           1 error

Diagnostics:
  kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release> (signalfx-agent):
    error: Object 'Kind' is missing in 'null'

Hi 👋 Just updated a few dependencies today.. including

@pulumi/kubernetes

from

3.4.1

to

3.7.2

, and it's just started consistently getting 403's for a couple of

ConfigFile

resources:

Error: Error fetching YAML file '<https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml>': 403 Forbidden
    Error: Error fetching YAML file '<https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml>': 403 Forbidden

Any ideas why this might've started happening? The error call stack just points at this line of code. I can't see anything obvious that could cause it. Rolling back to

3.4.1

everything works again 🤯

Hey everyone! On implementing this https://github.com/pulumi/examples/tree/master/kubernetes-py-jenkins, getting the following issue. Any suggestions on how I could resolve this?

@sparse-park-68967 do you have a roadmap / timeline for the helm releases preview having issues addressed and going stable? Basically we're wondering if we should wait for it before going to production. Bearing in mind if we go ahead with helm chart resources then we'll need to tear them down before switching to helm releases; unless I'm missing something and this can be blue green somehow?

Is there a way to import all resources created by https://www.pulumi.com/docs/reference/pkg/kubernetes/yaml/configfile/ ? I've already applied the yaml file by hand and would now like pulumi to manage it.

Have you set up a kube config file on your gitlab runner that understands how to use

GOOGLE_CREDENTIALS

to connect to your cluster?

Why kubeconfig from my EKS cluster stop working?

How do I fix this?

It used to work fine

pulumi up

is just hanging indefinitely when trying to create

k8s.yaml.ConfigFile

for some CRD manifests in a directory.

k8s.yaml.ConfigGroup

is the same. Commenting out the block avoids the hang and works fine. They aren't mega CRDs, the biggest is ~550 lines. The code in question...

// Traefik CRDs
const crdDirectory = path.resolve(__dirname, "./crds");
fs.readdirSync(crdDirectory).forEach((filename) => {
  new k8s.yaml.ConfigFile(
    `crd-${filename}`,
    {
      file: `${crdDirectory}/${filename}`,
    },
    {
      provider: args.provider,
      parent: this,
    }
  );
});

Versions:

"@pulumi/kubernetes": "^3.7.2",
"@pulumi/pulumi": "^3.13.0",

I have another project doing the same thing that I haven't touched in a while, using @pulumi/pulumi v3.0.0 and @pulumi/kubernetes v3.5.0, which was working fine. Downgrading to those same versions for this still hangs unfortunately.

is it normal for ConfigMaps to being replaced if you change the content?

Re helm releases, was there also a change in behavior from helm charts re specifying a namespace? For example, installing the fluent bit chart via helm chart resource, all k8s resources were created in the specified namespace (

kube-system

). However, installing the fluent bit chart via helm release resource, all k8s resources were created in the

default

namespace. Since the fluent bit chart does not use

{{ .Release.Namespace }}

in the chart to control the namespace, I presume the helm chart resource was manipulating the namespace directly? This area is a bit of a mess it seems - see https://github.com/helm/helm/issues/5465#issuecomment-692133971 From a chart user's pov, I just want to override the namespace as before, so based on the following from the above comment:

After deliberating over the use cases described earlier, I am certain they can be solved in different manners than the proposal described earlier; either by the tooling they're using (flux and spinnaker allow you to specify the namespace), by injecting the metadata yourself through the templates, or by using the post-renderer introduced in Helm 3.1.

Is it reasonable that pulumi should be like flux and spinnaker and allow us to override the namespace?

@here I am trying to create mapping type custom resource using pulumi v2 k8 SDK, below is the code snippet , but when i see debug logs of pulumi i see Spec object going as empty, not able to figure out whats goiing on, please help #kubernetes #golang #pulumi-kubernetes-operator

func CreateCustomResource(ctx *pulumi.Context, applicationName string, apiKey string, urlPrefix string, apiVersion string, kind string, labels pulumi.StringMap, timeout int64,k8sProvider *k8s.Provider,serviceAccount *v1.ServiceAccount)(*k8sApiExt.CustomResource, error){


   type SpecType struct {
      Prefix pulumi.String
      Service pulumi.String
      Headers pulumi.StringMap
      TimeOut <http://pulumi.Int|pulumi.Int>
   }

   //headersData := pulumi.StringMap{"x-api-key":pulumi.String(apiKey)}

   spec := SpecType{
      Prefix:  pulumi.String(fmt.Sprintf("/%s/",urlPrefix)),
      Service: pulumi.String(applicationName),
      TimeOut: <http://pulumi.Int|pulumi.Int>(timeout),
   }

   var otherfields k8s.UntypedArgs
   otherfields = make(k8s.UntypedArgs, 1)
   otherfields["Spec"] = spec


   customResourceDefnArgs := k8sApiExt.CustomResourceArgs{
      ApiVersion: pulumi.String(apiVersion),
      Kind:       pulumi.String(kind),
      Metadata:   metav1.ObjectMetaArgs{
         Labels:                     labels,
         Name:                       pulumi.String(applicationName),
         Namespace:                  pulumi.String(applicationName),

      },

      OtherFields: otherfields,

   }
   data, err := json.Marshal(customResourceDefnArgs)
   if(err == nil){
      logger.Infof("datataa= %s-%s",data,apiKey)
   }

   return k8sApiExt.NewCustomResource(ctx,GetResourceName(applicationName,"mapping"),&customResourceDefnArgs,pulumi.DependsOn([]pulumi.Resource{serviceAccount}),pulumi.Provider(k8sProvider))


}

Since deploying the Datadog helm chart, I get the following

pulumi preview

diff even if nothing has changed. This ultimately causes my datadog pods to get recreated, which is not desirable. I am deploying to AKS K8s (1.21.2) using Pulumi 3.12.0

pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:preprod::aks::pulumi:pulumi:Stack::aks-preprod]
            [provider=urn:pulumi:preprod::aks::pulumi:providers:kubernetes::k8s-provider::046e67e2-9780-4010-9c77-9999999ebefd]
          ~ spec: {
              ~ template: {
                  ~ metadata: {
                      ~ annotations: {
                          ~ checksum/clusteragent_token: "32a656c3c7aeb06e5c36xxx" => "8027d4026d2e72484f1xxx"
                        }
                    }
                }
            }
        +-kubernetes:core/v1:Secret: (replace)
            [id=default/datadog-chart-cluster-agent]
            [urn=urn:pulumi:preprod::aks::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/datadog-chart-cluster-agent|helm.sh/v3:Chart$kubernetes:core/v1:Secret::default/datadog-chart-cluster-agent>]
            [provider=urn:pulumi:preprod::aks::pulumi:providers:kubernetes::k8s-provider::046e67e2-9780-4010-9c77-9999999ebefd]
          ~ data: {
            }
        ~ kubernetes:apps/v1:DaemonSet: (update)
            [id=default/datadog-chart]
            [urn=urn:pulumi:preprod::aks::kubernetes:<http://helm.sh/v3:Chart$kubernetes:apps/v1:DaemonSet::default/datadog-chart|helm.sh/v3:Chart$kubernetes:apps/v1:DaemonSet::default/datadog-chart>]
            [provider=urn:pulumi:preprod::aks::pulumi:providers:kubernetes::k8s-provider::046e67e2-9780-4010-9c77-9999999ebefd]
          ~ spec: {
              ~ template: {
                  ~ metadata: {
                      ~ annotations: {
                          ~ checksum/clusteragent_token: "bcc328b0b69baa07a7fae32a6baxxx" => "5200e78e7733904901f9511a094e8xxx"
                        }
                    }
                }
Resources:
    ~ 2 to update
    +-1 to replace
    3 changes. 103 unchanged

And the pruned logs from the

pulumi up

look like this:

-- kubernetes:core/v1:Secret default/datadog-chart-cluster-agent deleting original 
 ~  kubernetes:apps/v1:Deployment default/datadog-chart-cluster-agent updating [diff: ~spec]
 ~  kubernetes:apps/v1:DaemonSet default/datadog-chart updating [diff: ~spec]
 -- kubernetes:core/v1:Secret default/datadog-chart-cluster-agent deleting original 
 -- kubernetes:core/v1:Secret default/datadog-chart-cluster-agent deleted original
 ~  kubernetes:apps/v1:Deployment default/datadog-chart-cluster-agent updating [diff: ~spec]; [1/2] Waiting for app ReplicaSet be marked available
 ~  kubernetes:apps/v1:Deployment default/datadog-chart-cluster-agent updating [diff: ~spec]; Deployment initialization complete
 ~  kubernetes:apps/v1:Deployment default/datadog-chart-cluster-agent updated [diff: ~spec]; Deployment initialization complete 
 ~  kubernetes:apps/v1:DaemonSet default/datadog-chart updated [diff: ~spec] 
 +- kubernetes:core/v1:Secret default/datadog-chart-cluster-agent replacing [diff: ~data];
 +- kubernetes:core/v1:Secret default/datadog-chart-cluster-agent replaced [diff: ~data]; 
 ++ kubernetes:core/v1:Secret default/datadog-chart-cluster-agent creating replacement [diff: ~data]; 
 ++ kubernetes:core/v1:Secret default/datadog-chart-cluster-agent creating replacement [diff: ~data]; 
 ++ kubernetes:core/v1:Secret default/datadog-chart-cluster-agent created replacement [diff: ~data];

Can someone help point me in the right direction?

@sparse-park-68967 I think there's an issue with the new helm releases, their dependencies and uninstall / delete AFAICT, under the hood the call to helm uninstall does not wait, so dependencies do not unwind in the correct order. When this deletes a resource with a finalizer then other resources are orphaned and it cascades from there...

PS. helm releases are much faster than helm charts - shaves ~15-20m off times to create and destroy the k8s stack Presumably due to fewer pulumi api roundtrips for hundreds of fewer resources since helm releases does not look through and enumerate chart resources

Hi Team, does pulumi kubernetes provider allows us to do 

exec

  as we can do in terraform provider? https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#exec-plugins #kubernetes

Moving it from #general to #kubernetes Does anyone know an example of installing and configuring kubernetes "cluster-autoscaler" to use ManagedNodeGroup with pulumi/eks and pulumi/kubernetesx?

Hi Team, I'm using rancher2 and kubernetes plugins for setting up my cluster. below is how my python looks,

rancher_ci_cluster = rancher2.Cluster()

k8s_provider = k8s.Provider("cluster-access",
    kubeconfig=rancher_ci_cluster.kube_config
)

sa = k8s.core.v1.ServiceAccount("microservice",
    metadata=k8s.meta.v1.ObjectMetaArgs(
      namespace="monitoring",
      name="microservice"
    )
)

this keep throwing below error

error: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://992E92B1DDDD40F85E620599B78C3F9C.sk1.us-east-1.eks.amazonaws.com/openapi/v2?timeout=32s>": dial tcp: lookup <http://992E92B1DDDD40F85E620599B78C3F9C.sk1.us-east-1.eks.amazonaws.com|992E92B1DDDD40F85E620599B78C3F9C.sk1.us-east-1.eks.amazonaws.com> on 10.0.4.55:53: no such host

not sure if the provider is getting set correctly or not, can someone help me here please? p.s: that's not my cluster api server endpoint too

Hello, wonder if there is any way to get more visibility on https://github.com/pulumi/pulumi/issues/4996 . It stops us from automating destroy/rebuild of clusters

Hey! Any1 able to help me here? https://stackoverflow.com/questions/69437619/how-to-modify-the-aws-auth-config-map-after-creating-eks-with-pulumi

It appears that changing an Helm release

values

(Go automation API) doesn’t trigger an update during Up() operations. I can see the new value in the stack state file but not when I run

helm get values <release>

. I have tried setting Replace=true / ForceUpdate=true / ResetValues=true but no luck. Am I doing something wrong?

Hi, how can I use Pulumi to get istio configmap and to update the data? Istio configmap is created by

istioctl

, rather than Pulumi itself I know for resources that created by pulumi, I can definitely update that by running

pulumi up

, but how about resource that not created by

pulumi

?

Hi, is there any setting in pulumi that makes a Job Resource run on each deployment?