https://pulumi.com logo
Join the conversationJoin Slack
Channels
announcements
automation-api
aws
azure
blog-posts
built-with-pulumi
cloudengineering
cloudengineering-support
content-share
contribex
contribute
docs
dotnet
finops
general
getting-started
gitlab
golang
google-cloud
hackathon-03-19-2020
hacktoberfest
install
java
jobs
kubernetes
learn-pulumi-events
linen
localstack
multi-language-hackathon
office-hours
oracle-cloud-infrastructure
plugin-framework
pulumi-cdk
pulumi-crosscode
pulumi-deployments
pulumi-kubernetes-operator
pulumi-service
pulumiverse
python
registry
status
testingtesting123
testingtesting321
typescript
welcome
workshops
yaml
Powered by Linen
kubernetes
  • f

    fast-easter-23401

    03/18/2022, 5:27 PM
    Hello folks, We’re using secrets to store some sensitive data (namely serviceAccounts keys). I was wondering what would happen to pods (or workloads more generally) using those secrets when we rotate our serviceAccount keys. Pulumi would then try to recreate a new secret with new values. Will these be propagated to the workloads using them? Many thanks,
    q
    • 2
    • 1
  • f

    famous-salesclerk-74711

    03/18/2022, 11:16 PM
    has anyone seen spurious
    urn
    name conflicts when creating a brand new resource…where there DEFINITELY isn’t an existing resource with that name?
    q
    • 2
    • 2
  • e

    echoing-angle-67526

    03/18/2022, 11:51 PM
    is there a way to deploy to k8s and generate the yaml to a directory at the same time? currently you need to create a separate k8s provider and set the renderYamlToDirectory property as setting this property causes the other attributes like kubeconfig to be ignored. i could create two k8s objects with different providers, since you can only pass one provider, but it would be nice if you could pass multiple providers to k8s objects.
    c
    • 2
    • 8
  • b

    billowy-horse-79629

    03/23/2022, 10:57 AM
    Hey guys, I’m trying to create a Helm Release on an already exist cluster (managed by Pulumi). For some reason i’m getting this error on my
    pulumi up
    action on Github Actions:
    +  kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release> eks-bootstrap creating error: can't create Helm Release with unreachable cluster: unable to load Kubernetes client configuration from kubeconfig file: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
    Here is the code i’m creating my Helm Release (occurs after the eks k8s cluster creation)
    export function createHelmBootstrapChart(stackName: string) {
    
        const releaseArgs: k8s.helm.v3.ReleaseArgs = {
            atomic: true,
            name: "eks-bootstrap",
            chart: "./charts/eks-bootstrap-chart",
            dependencyUpdate: true,
            namespace: "kube-system",
            valueYamlFiles: [new FileAsset(`./charts/eks-bootstrap-chart/values-${stackName}.yaml`)]
            
        }
        new k8s.helm.v3.Release("eks-bootstrap", releaseArgs)
    
    }
    Do you guys know how to set the Kubeconfig ? because it looks like the kubeconfig is missing during runtime on the github action.
    • 1
    • 2
  • b

    billowy-horse-79629

    03/23/2022, 1:10 PM
    Does anyone know how to handle updates in Helm Chart/Release ? I created a release using Pulumi but it never updates when i’m changing the chart or the pulumi code. For example, I changed the release namespace, it updates the Pulumi state but doesn’t create the release in the new namespace (neither delete the old release). Any hints with that ?
    ➕ 2
  • b

    boundless-telephone-75738

    03/23/2022, 10:10 PM
    So, I've got about 160 helm releases in a stack connecting to an EKS cluster on aws. Trying to run it locally is now actually timing out telling me my upgrade token has expired. I suspect trying to build the state for my helm releases is being throttled by aws. Any advice, should I split my stack into substacks so that I don't have to poll quite as many for each op?
  • c

    crooked-pillow-11944

    03/23/2022, 10:17 PM
    Yes
    :thank-you: 1
  • b

    best-state-21371

    03/24/2022, 4:36 PM
    So I have a general k8s question, I have a Saas product that we are looking at modernizing in a lot of areas. It is a CRM for nonprofits that has 500-1k unique customers. We have them in a monolith app now with 1 shared database. We are looking at increasing isolation, improving scaling, noisy neighbor issues, etc. I have been told by a few that it would make sense to have a structure where each of our customer’s have their own name space to get better isolation. But I am curious if this is the best pattern? Is this what most people are doing now for multi-tenant apps?
    c
    • 2
    • 1
  • b

    busy-journalist-6936

    03/24/2022, 7:18 PM
    I could use some help solving how to write kustomize transformations to remove a namespace object from being generated by @pulumi/kubernetes's kubernetes.kustomize.Directory where the namespace object is coming in from an upstream that is out of my control. Problem of kustomize generating the namespace object results in:
    Updating (keycloak):
         Type                                                  Name                   Status                  Info
         pulumi:pulumi:Stack                                   tkl-keycloak-keycloak  **failed**              1 error
         └─ kubernetes:kustomize:Directory                     keycloak                                       
     +      ├─ kubernetes:core/v1:Namespace                    keycloak               **creating failed**     1 error
    q
    l
    • 3
    • 6
  • b

    big-potato-91793

    03/24/2022, 9:10 PM
    Got a weird error when trying to used my crd generated with crd2pulumi. Anyone has already that kind of a problem?
    TypeError: Invalid Version:
            at new SemVer (/builds/${PROJECT}/pulumi/node_modules/semver/semver.js:332:11)
            at compare (/builds/${PROJECT}/pulumi/node_modules/semver/semver.js:647:10)
            at Function.eq (/builds/${PROJECT}/pulumi/node_modules/semver/semver.js:693:10)
            at sameVersion (/builds/${PROJECT}/pulumi/node_modules/@pulumi/runtime/rpc.ts:703:57)
            at register (/builds/${PROJECT}/pulumi/node_modules/@pulumi/runtime/rpc.ts:718:17)
            at Object.registerResourcePackage (/builds/${PROJECT}/pulumi/node_modules/@pulumi/runtime/rpc.ts:780:5)
            at Object.<anonymous> (/builds/${PROJECT}/pulumi/node_modules/@ticketmaster/pipelog-crds/build/index.js:25:16)
            at Module._compile (node:internal/modules/cjs/loader:1103:14)
            at Object.Module._extensions..js (node:internal/modules/cjs/loader:1155:10)
            at Module.load (node:internal/modules/cjs/loader:981:32)
  • r

    rhythmic-whale-48997

    03/25/2022, 9:55 AM
    I'm trying to use Command and run something on the create. There is already an EKS cluster and I have it's kubeconfig From the example on https://www.pulumi.com/registry/packages/command/#graceful-cleanup-of-workloads-in-a-kubernetes-cluster I'm getting an error
    environment[KUBECONFIG] value: An error occurred decoding 'command.environment[KUBECONFIG] value': Cannot decode Object{} to type string; it isn't a struct, and no custom decoder exists
    Is there some better way to provide kubeconfig to the command and I would like to avoid having it on filesystem.
    q
    • 2
    • 3
  • f

    freezing-quill-32178

    03/28/2022, 8:48 AM
    Hello everyone, happy Monday! Maybe a silly question but how do you get the dynamic resource names created by
    new k8s.helm.v3.Release(…)…
    for
    XYZ
    Helm chart… if you don’t specify the
    name
    and leave Pulumi to autogenerate it, or even specify the
    name
    whichever… and use the resource names later on it the program without inferring the name based on the prefix and whatever the chart is?
    b
    • 2
    • 7
  • b

    breezy-bear-50708

    03/28/2022, 2:11 PM
    How are people dealing with existing/managed resources? We have a GKE cluster and want to modify some of the resources GKE creates, like the
    standard
    StorageClass
    . Haven't been able to find a good, automated way to do this.
    import
    requires the resource fields match exactly so it's not particularly helpful without lots of manual work. Currently we're using a dynamic provider to patch the resource but it's pretty flaky.
    w
    • 2
    • 3
  • s

    sparse-apartment-71989

    03/28/2022, 11:45 PM
    Is there a way to get a fixed Secret name from Pulumi rather than a name with a hash value appended to the end of the name I specify?
    const dockerSecret = new Secret('docker-hub-credentials',
      {
        data: {
          'username': docker_username, // Base64-encoded value
          'password': docker_password  // Base64-encoded value
        }
      },
      {
        provider: k3sProvider
      }
    )
    In the above example, the secret created is docker-hub-credentials-m4vtegxw. In other, non-Pulumi programs, I have to remember to update the referenced secret name to the full Pulumi-generated name value. I’d expected to be able to set a name property that would be analogous to setting a
    name
    property on an S3 Bucket to fix the name.
    b
    • 2
    • 1
  • b

    billowy-horse-79629

    03/29/2022, 9:18 AM
    If anyone have an idea what’s happening here… https://pulumi-community.slack.com/archives/C84L4E3N1/p1648488781552619
    s
    • 2
    • 3
  • a

    adorable-action-51248

    04/02/2022, 7:56 AM
    I have an issues with helm release. it seems it doesnt detect changes in the chart directory. while
    helm upgrade
    does.
    • 1
    • 1
  • c

    chilly-plastic-75584

    04/04/2022, 9:23 PM
    Handling Partial Failures/Timeouts With Kubernetes and Pending Operations Just got my first prod kubernetes deployment out the door. 🎉 Now I'm running into a situation and not sure if it's easily resolved. If I deploy to a new environment and run into it failing, things can get into a stuck state. For example, Ingress timed out. Now it isn't able to proceed because it tries to recreate the ingress that was already created, but not marked as healthy. I've adjusted things to have a longer timeout, but that doesn't fix the possible failure I'd still deal with in the future. • I can't access the cluster to try and import and I don't see any import options with kubernetes object • Only thing I can see is to run a destroy on stack and redeploy but that's not ideal since everything is up except for pulumi knowing the ingress was successfully created. Impacts the downtime. • Right now I had to ask the admin to delete the ingress manually and that won't fly in any future deployments. How do I handle this? In this scenario the state track is actually causing a problem for me. Also since state tracking is involved it's causing others to question the approach since kubectl apply and such don't do state tracking. I want to avoid having to shelve this work but have to figure out some way to handle this myself in the future. Would appreciate any guidance on how I can stablize this with the limited access I have.
    r
    g
    • 3
    • 8
  • c

    curved-morning-41391

    04/05/2022, 10:14 PM
    Hello, has there been any thought into having Kubernetes as the backing state store? It would be nice if I didn't need another source of truth for my k8s objects
  • c

    cuddly-football-51817

    04/06/2022, 11:35 AM
    Hello, I have been trying to deploy Argo-CD via Pulumi. Seems due to CRD issue, its not possible to directly deploy it. However i was trying to use the argocd-autopilot to bootstrap my rest of the cluster. I am planning to run the argocd-autopilot via kubernetes job where i can pass the kubeconfig as well as git PAT token to the job to bootstrap the argocd and the cluster. However i am wandering if there is any way i can run another kubernetes job when i am destroy the stack which does removes the argo apps and cleans up the cluster?
  • g

    great-sunset-355

    04/06/2022, 1:32 PM
    Hi, I'm trying to deploy CSI Store chart https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/ But I do get this warning - this issue mentioned is closed however I cannot deploy the chart successfully
    warning: This resource contains Helm hooks that are not currently supported by Pulumi. The resource will be created, but any hooks will not be executed. Hooks support is tracked at <https://github.com/pulumi/pulumi-kubernetes/issues/555> -- This warning can be disabled by setting the PULUMI_K8S_SUPPRESS_HELM_HOOK_WARNINGS environment variable
    kubernetes  resource  3.18.0   77 MB   n/a        22 minutes ago
    b
    • 2
    • 4
  • m

    most-lighter-95902

    04/06/2022, 4:00 PM
    Just created a cluster with Pulumi’s
    eks
    package and it doesn’t seem to install the latest Kubernetes version?
  • m

    most-lighter-95902

    04/06/2022, 4:01 PM
    EKS now has v1.22 available for Kubernetes but Pulumi’s
    eks
    package doesn’t seem to find it on new install?
  • m

    most-lighter-95902

    04/06/2022, 4:01 PM
    Should I specify the version explicitly?
    b
    • 2
    • 5
  • c

    crooked-pillow-11944

    04/06/2022, 4:08 PM
    You should always specify the version explicitly
    ➕ 2
  • b

    bulky-cat-1703

    04/06/2022, 5:45 PM
    Hey all, just joined the chat. Currently the kubernetes provider seems to be chugging pretty hard when it comes to large CRDs (Deploying argocd hangs forever). Any recommendations? I’ve filed an issue in the relevant repo. Are there any mitigations atm?
    b
    c
    • 3
    • 6
  • n

    nutritious-petabyte-61303

    04/07/2022, 1:18 AM
    Is it possible to render kubernetes YAML with Pulumi?
  • n

    nutritious-petabyte-61303

    04/07/2022, 1:19 AM
    seems so
  • n

    nutritious-petabyte-61303

    04/07/2022, 1:19 AM
    https://www.pulumi.com/blog/kubernetes-yaml-generation/
  • n

    nutritious-petabyte-61303

    04/07/2022, 1:20 AM
    interestingly enough I've found that using Pulumi for the deployments themselves is often less than ideal, as it handles some of the state updates wrong or fails to do cleanup appropriately in failure scenarios, figured rednering to yaml and using a different deployment tool would work better
    l
    w
    +2
    • 5
    • 10
  • k

    kind-room-82948

    04/14/2022, 9:32 PM
    I was wondering if anyone has any methods for graceful rollbacks of failed Kubernetes deployments. If we update one of our deployment images to a new version and it doesn't roll out successfully we would prefer if that change was just automatically rolled back to the previous good state. Is the current solution just to do a revert commit and apply again?
    b
    n
    • 3
    • 31
Powered by Linen
Title
k

kind-room-82948

04/14/2022, 9:32 PM
I was wondering if anyone has any methods for graceful rollbacks of failed Kubernetes deployments. If we update one of our deployment images to a new version and it doesn't roll out successfully we would prefer if that change was just automatically rolled back to the previous good state. Is the current solution just to do a revert commit and apply again?
b

billowy-army-68599

04/14/2022, 9:42 PM
when you say:
we would prefer if that change was just automatically rolled back to the previous good state
Do you mean the Pulumi side of things, or the Kubernetes side of things? pulumi's await logic will mean that the old deployment itself will still be running
k

kind-room-82948

04/14/2022, 9:43 PM
Probably both would be preferred, but in k8s at least it will have both the old pods and the new pods that are going to be stuck crashing
If it ends up failing the pulumi checks (so say if the new pods never get ready or are crashing), our preference would be for it to go back to the known good state in both pulumi and in the k8s objects
n

nutritious-petabyte-61303

04/18/2022, 9:09 PM
there are a few k8s operators that handle things like this, perhaps you could look at knative or argocd
my impression is that the proper way to approach this kind of a solution is to use a k8s operator that handles it
that being said, I'm sure you could automate it with strictly pulumi too, it's just slightly more risky as you'd be making the deployment an imperative process instead of declarative (what happens if the deployment runner crashes after a failed deploy but before cleanup?), so at the very least you'd need to ensure it's idempotent
b

billowy-army-68599

04/18/2022, 9:15 PM
@nutritious-petabyte-61303 I'm not sure what you're getting at here, Pulumi is declarative and idempotent
n

nutritious-petabyte-61303

04/18/2022, 9:23 PM
pulumi is declarative but the deployment is not
that being said, pulumi also has a k8s operator you could use: https://www.pulumi.com/docs/guides/continuous-delivery/pulumi-kubernetes-operator/
I'm not sure how idempotent pulumi truly is, but I know there are (or have been) some scenarios where it doesn't do k8s cleanup appropriately. Essentially pulumi could try applying a resource (inserting it to the cluster state but not pulumi state) and fail to deploy it (without cleaning it up from the k8s state as it was never inserted into the pulumi state)
b

billowy-army-68599

04/18/2022, 9:26 PM
Again I'm not following, what do you mean the deployment is not? If that's happening, it's a bug and we need to fix it
n

nutritious-petabyte-61303

04/18/2022, 9:26 PM
Not sure if that was clear, but I've hit a few scenarios where there's drift between the actual k8s state and the state tracked by pulumi
b

billowy-army-68599

04/18/2022, 9:27 PM
Okay, if that's happening it's a bug. Please file issue and let me know immediately, this is the first I'm hearing of it
n

nutritious-petabyte-61303

04/18/2022, 9:27 PM
Deployment is imperative because it's handled in steps, (i.e.
wait for resource X to deploy before deploying resource Y
). This is unavoidable in most scenarios, but in the context of k8s it might not make as much sense
I'd consider the deployment declarative if the pulumi operator was installed in the cluster, and the deployment would amount to "deploy this version of the pulumi program using the operator"
Most commonly I see the client running the imperative deployment process however, which means a failure during it (i.e. network issue) can leave the deployment in a weird state
Okay, if that's happening it's a bug. Please file issue and let me know immediately, this is the first I'm hearing of it
It's been a few months since I last hit the issue and I no longer remember what it was specifically, but I'll be sure to do so the next time it happens.
As a rule of thumb however, I suspect if pulumi applies any resource pushed to the k8s state also on its own state immediately before healthchecks, there wouldn't be any issue, as that represents what actually happens during a deploy (once a resource is pushed on k8s state, it exists there regardless if it fails or succeeds).
The disparities I've hit have always been caused by pulumi running healthchecks before updating its internal state to match what was pushed to k8s, and in some scenarios losing track of what was pushed (because it failed to deploy)
Also to be clear, it's been a few months since the last time I've had a broken deploy like this, so for all I know it might already be fixed.
k

kind-room-82948

04/18/2022, 9:33 PM
I'm not sure this is exactly what I'm looking for, the operator seems more of a tool for running pulumi within kubernetes natively. It doesn't really seem to affect the actual object control
b

billowy-army-68599

04/18/2022, 9:33 PM
I think we have a different definition of imperative and declarative. My definition of declarative is that a DAG is evaluated and then it's not possible to manipulate that evaluation. It seems your version of declarative is focused on the Kubernetes reconciliation model. The fact we check for successful healthchecks doesn't change the declarative nature of Pulumi at all and you can even turn it off with a single flag in the provider
n

nutritious-petabyte-61303

04/18/2022, 9:34 PM
you'd be looking for a canary deployment using a k8s operator that can handle the rollback if it needs to be done, or some pulumi magic I'm not familiar with. I'm fairly certain Knative or ArgoCD operators would work for that (though Knative is overkill)
@billowy-army-68599 yeah, pulumi is declarative from a programming-model standpoint, but the deployment process is not declarative, if that makes sense
(well it entirely depends on how you deploy, since the operator does exist)
generally anything that needs to track intermediate state can not be declarative IMO
b

billowy-army-68599

04/18/2022, 9:36 PM
no, it doesn't make sense i'm afraid, We seem to disagree on this, but that's okay
👍 1
n

nutritious-petabyte-61303

04/18/2022, 9:36 PM
the problem arises from the dependency on previous state
that makes it imperative
b

billowy-army-68599

04/18/2022, 9:37 PM
i respectfully disagree, and we'll leave it at that
n

nutritious-petabyte-61303

04/18/2022, 9:38 PM
yeah we're going a bit offtopic here debating what is or isn't imperative or declarative 😄
View count: 10