Hi everyone. I'm currently debating the merits of managing K8s deployments in Pulumi vs. Helm. I already provision the K8s infrastructure (cluster, etc), but have not yet embraces making K8s API calls (to deploy, etc) from within Pulumi. Am I right in assuming that Pulumi can be a full replacement for Helm -- for example, it can compute the diffs in desired/deployed K8s resources and make the necessary changes (such as deleting a Service if you've deleted it's definition in the Pulumi code). I do like the idea of colocating infra provisioning with app deployment, but I do worry about the ability for a corruption in state file due to some bad/unlucky infra provisioning preventing progress in making application deployment changes inside K8s. Does anyone have thoughts on how to weigh the pros and cons here?

ive created a custom ComponentResource of which a child resource is a helm chart (ingress-nginx specifically). when using this ComponentResource in a stack, it does not appear that changes to helm config result in changes to changes to resources created by the helm chart. does anyone have thoughts on this?

I’m creating an apps/v1:Deployment. Something is wrong in my spec I think, but Pulumi is masking my entire spec as a Secret making it hard to debug? Is there something causing this? I don’t think I’m even using a secret as an input to the Spec? eg,

+ kubernetes:apps/v1:Deployment: (create)
        [urn=urn:pulumi:sandbox-usw2a::app-60-app::kubernetes:apps/v1:Deployment::be-innkeeper-deployment]
        [provider=urn:pulumi:sandbox-usw2a::app-60-app::pulumi:providers:kubernetes::k8s::15e3e6ce-24ca-419c-a449-3238c4372aa6]
        apiVersion: "apps/v1"
        kind      : "Deployment"
        metadata  : {
            labels   : {
                <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: "pulumi"
            }
            name     : "be-innkeeper"
            namespace: "default"
        }
        spec      : "[secret]"

anyone here know how to read in a K8s entity that isn't managed by Pulumi? I need to annotate an EKS provided service account programatically

but the only methods I see are

corev1.GetServiceAccount

which requires a Pulumi ID

Hey Guys Good Morning, I am deploying a helm chart with pulumi and every service name got an extra hash value at the end. How can i disable this auto naming with pulumi ?

Is there a way to find Helm created resource other than this?

wordpress.getResourceProperty("v1/Service", "wpdev-wordpress", "status");

This approach doesn't work when some charts like kube-prometheus truncates the long resource names. For example chart creates two different service names based on pulumi resource name.

new k8s.helm.v3.Chart(
      `kube-prometheus-example`,
      ...
)

new k8s.helm.v3.Chart(
      `kube-prometheus-example1`,
      ...
)

generates following Services..

kube-prometheus-example-kube-alertmanager
kube-prometheus-example1-kub-alertmanager

In such cases I cannot lookup the service its created..

I'm trying to install the kube prometheus stack via its helm chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack ... I've seen a few people mention it in the slack channels. What transformations are you using to get it to work with pulumi?

I am trying to install kube-prometheus-stack via its helm chart. It creates a couple of crds and I want to create some resources using those crds as well. What’s happening is, the creation of these resources using those crds fails (error - resource of type doesn’t exist) because it somehow executes before the kube-prometheus-stack has finished installing. The kube-prometheus-stack creation and the resources are all in different files. I tried adding

dependsOn

to the custom resources but that does not help. Is there any example where I see dependent k8s resources being created from different files. Also, I am using typescript. Any help is highly appreciated.

Hi, today I updated the "values" part of a helm chart (v3) I deployed with pulumi, but pulumi didn't notice any change, therefore reported "everything unchanged". Is this a known limitation or should I dig deeper to (maybe) reveal an issue?

Pulumi works well with modern “immutable infrastructure” architectures, where bootstrapping and patching are unnecessary. In such cases, configuration management is not needed in the usual sense.

hi! how does immutable infra works with persistent storage in k8s? please see this example: i have deployed a statefulset application on k8s, whose pods and persistent volume live on a node(machine). now i am going to do some system update to that machine(node). for “immutable infrastructure”, it will destroy the old one, which will definitely leave my app in error state. i believe ppl who designed "immutable infra" already figured a elegant solution for the above. can someone share your thoughts here? thanks!

You'd need to create a new node pool and do a blue/green type of upgrade. This isn't something an Infrastructure-as-code solution will do for you

I'm assuming most of us are using Server Side Apply at this point. How do I configure the manager name of fields managed by pulumi?

just curious, for ppl who use pulumi rke provider, if you update some config/parameter of a existing cluster, the cluster will get updated in place right? it won't be destroyed and recreated

I am attempting to use the pulumi_tls module to populate a k8s secret how do I Base64 encode the output of the tls module for use in secrets?

Hello folks, I have a quick question about how to install Grafana using its Helm chart. What I'm trying to do is to install Grafana with a datasource, but it fails to read the configuration when using Pulumi's

k8s.helm.v3.Chart

values

parameter. I suspect it happens because one of the keys is

datasources.yaml

which contains a

.

and may be expanded to something the helm chart does not understand. This YAML works when installing the chart not using Pulumi:

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: "Cinnamon Prometheus"
        type: prometheus
        access: proxy
        url: <http://prometheus-server.default.svc.cluster.local>
        editable: true

But it fails when using Pulumi (typescript api):

new k8s.helm.v3.Chart(
  "grafana",
  {
    chart: "grafana",
    fetchOpts: {
      repo: "<https://grafana.github.io/helm-charts>",
    },
    values: {
      datasources: {
        "datasources.yaml": {
          apiVersion: 1,
          datasources: {
            name: "Cinnamon Prometheus",
            type: "prometheus",
            access: "proxy",
            url: "<http://prometheus-server.default.svc.cluster.local>",
            editable: true,
          },
        },
      },
    },
  },
  { provider: clusterProvider },
);

How would one scale the PV/PVC of a statefulset deployed via helm/pulumi? Changing the storage value in the statefulset is ignored (not even a replace which is probably a good idea :))

is kubernetesx still maintained? Some fairly simple things (e.g. a pvc.mount() in both an init container and main container) give naming collisions.

Has anyone used https://github.com/benesch/pulumi-kubernetes-proxy ? Is there a better-way to access in-vpc/cluster resources like a postgres? I would like to provision another role & db…..

I have a situation I’m not quite sure what’s causing it. I have the following update diff if I do preview/up:

+-kubernetes:core/v1:Secret: (replace)
        [id=itay9/odas-secrets-o46s05yb]
        [urn=urn:pulumi:tenant-itay9::okera-trial-tenants::kubernetes:core/v1:Secret::odas-secrets]
        [provider=urn:pulumi:tenant-itay9::okera-trial-tenants::pulumi:providers:kubernetes::k8s-ssa-provider::b438617b-1a9e-4bd4-94f4-9ab2a01529a2]
      ~ stringData: {
          ~ SYSTEM_TOKEN: "[secret]" => "[secret]"
        }

In the code, the secret is created like this:

...

		StringData: pulumi.StringMap{
            ...,
			"SYSTEM_TOKEN":                 systemToken,

and

systemToken

is defined like this:

systemToken := pulumi.All(jwtKey.PrivateKeyPem).ApplyT(
		func(args []interface{}) (string, error) {
           ...
		},
	).(pulumi.StringOutput)

This would all make sense if there was an update in

systemToken

, but there isn’t - that’s not one of the resources that Pulumi is saying requires an update. Any idea why this might be the case? How do I debug something like this?

I’m wondering on using Security Groups for Pods on EKS — I’ve got most of it automated with a Managed Node Group + using the EKS Addons. — general instructions are here: https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html One spot I’ve ran into an issue is with patching the

ENABLE_POD_ENI

environment variable? Any ideas on how to best do this with Pulumi? The AWS documentation says to run ”

kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true

How do people manage the sometimes complex web of dependencies so that resource deletion works without timing out? ("timed out waiting to be Ready"); e.g. • install aws load balancer controller (albc) and kube prometheus stack (kps) via helm charts • configure kps values for ingress with alb annotations, so ingress creates an aws alb (and adds a finalizer) • kps effectively depends on albc and this must be explicit via pulumi otherwise you get the timeouts above Do people just carefully wire these dependencies explicitly, in which case how do you handle the inevitable circular dependencies?

I've reordered resources and taken dependencies on helm charts rather than just crds (in separate config groups) and destroy got much further but was hanging for ages until finally failing with:

Diagnostics:
  pulumi:pulumi:Stack (k8s-dev):
    could not get token: RequestError: send request failed
    caused by: Post <https://sts.amazonaws.com/>: dial tcp: lookup <http://sts.amazonaws.com|sts.amazonaws.com> on 192.168.65.5:53: no such host
    could not get token: RequestError: send request failed
    caused by: Post <https://sts.amazonaws.com/>: dial tcp: lookup <http://sts.amazonaws.com|sts.amazonaws.com> on 192.168.65.5:53: no such host

    error: update failed

  kubernetes:<http://apiextensions.k8s.io/v1:CustomResourceDefinition|apiextensions.k8s.io/v1:CustomResourceDefinition> (appmesh-system/appmesh-controller-selfsigned-issuer):
    error: Delete "https://[REDACTED].<http://gr7.us-west-2.eks.amazonaws.com/apis/cert-manager.io/v1alpha2/namespaces/appmesh-system/issuers/appmesh-controller-selfsigned-issuer|gr7.us-west-2.eks.amazonaws.com/apis/cert-manager.io/v1alpha2/namespaces/appmesh-system/issuers/appmesh-controller-selfsigned-issuer>": getting credentials: exec: executable
aws-iam-authenticator failed with exit code 1
    error: post-step event returned an error: failed to save snapshot: performing HTTP request: Patch "<https://api.pulumi.com/api/stacks/pharos/k8s/dev/update/27f96f11-ba00-4a38-b812-1a87d668ec8b/checkpoint>": dial tcp: lookup
<http://api.pulumi.com|api.pulumi.com> on 192.168.65.5:53: no such host

Resources:
    - 199 deleted

Duration: 2h44m7s

During the hang I checked resources and the aws appmesh controller and cert manager controllers had already been deleted. I didn't think it would matter since I didn't see any finalizers or owner refs associated with the

<http://cert-manager.io/v1/Issuer|cert-manager.io/v1/Issuer>

resource being deleted, but maybe I'm missing something? Looks like this is due to https://github.com/pulumi/pulumi-kubernetes/issues/861

So yeah.. back in single-cluster-duplicate-name-limbo.. A while ago I had to endure this: https://github.com/pulumi/pulumi/discussions/7382 Now I'm stuck with https://artifacthub.io/packages/helm/kong/kong where it complains that:

error: Duplicate resource URN 'urn:pulumi:prod.k8s.devsecops::baseline-k8s::app:k8s:foundation$kubernetes:<http://helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition::{uniquevar}-kongconsumers.configuration.konghq.com';|helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition::{uniquevar}-kongconsumers.configuration.konghq.com';> try giving it a unique name

where {uniquevar} is an alphanumeric string

anyone successfully use kong through helm with pulumi? This is a single cluster deployment so it should be unique regardless what I do..

So, I just ran

pulumi refresh

as I hadn’t used this repo/stack in a while and it seems to have untracked all of my k8s resources

and when I try to

pulumi preview

/

pulumi up

, it’s trying to create them from fresh

is there, uh, any way to undo this?

Hi - I have a strange issue where by when a deployment completes, the ingress temporarily returns 503 for services which are running - does anyone else see this? It clears after a few refreshes but that's obviously not ideal

Anyone noticed wrong ordering on delete? I make a PVC, then use the PVC in a deployment, creation works perfectly. But when deleting, it tires to delete the PVC first, which “stalls” because it is still mounted in the Deployment. Both the PVC and the deployment are in separate custom resources., the resource containing the PVC is passed intot he constructor of the custom resource that has the deployment.