What is interesting is that if we keep refreshing and upping, we eventually get past some ing’s errors and some will deploy. Sometimes only 1 out of the few expected ing’s will deploy while the rest error out.

Any help on this would be much appreciated! Thank you

while we are talking about aws=load-balancer-controller . . . when i deploy this manually with helm into my eks there is no problem, when i do so with pulumi :-

File "/Users/bmeehan/repos/itplat-pulumi-infrastructure/.venv/lib/python3.8/site-packages/pulumi_kubernetes/apiextensions/v1/CustomResourceDefinition.py", line 121, in __init__
        __self__._internal_init(resource_name, *args, **kwargs)
    TypeError: _internal_init() got an unexpected keyword argument 'status'
    error: an unhandled error occurred: Program exited with non-zero exit code: 1

details in thread, help please

A really really probably simple thing here.. Is there any way of making

k8s.yaml.ConfigFile/ConfigGroup

apply the resources to a namespace? I don't want to muck with the files, and they don't have a namespace declared in the yaml, so the transformation I've got seems to fail.. For instance:

const keyCloakAccounts = new k8s.yaml.ConfigGroup("keycloak-roles-accounts-bindings", {
                files: [
                    `${pathToKeyCloakOper}/role.yaml`, 
                    `${pathToKeyCloakOper}/role_binding.yaml`,
                    `${pathToKeyCloakOper}/service_account.yaml`
                ],
                transformations: [
                    (obj: any, opts: pulumi.CustomResourceOptions) => {
                        obj.meta.namespace = keyCloakNamespace.metadata.name;
                    }
                ]
            },{
                provider: args.providers.kubernetes,
                parent: this,
                dependsOn: keyCloakNamespace
            })

Which fails with:

TypeError: Cannot set property 'namespace' of undefined
        at Foundation.k8s.yaml.ConfigGroup.transformations (<path>\pulumi\baseline-k8s\components\kubernetes-components\foundation.ts:128:44)
        at parseYamlObject (<path>\pulumi\baseline-k8s\node_modules\@pulumi\yaml\yaml.ts:2925:9)
        at <path>\pulumi\baseline-k8s\node_modules\@pulumi\yaml\yaml.ts:2903:25
        at Array.map (<anonymous>)
        at <path>\pulumi\baseline-k8s\node_modules\@pulumi\yaml\yaml.ts:2903:14
        at runMicrotasks (<anonymous>)
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

I'm finding that if I use Pulumi to build a docker image and have a deployment/statefulset/etc reference that image, the deployment will update even before the image is built. If the image takes a while to build/push, this will guarantee a failed

pulumi up

. Has this been an issue for anyone else? Or should I file a bug ticket

I’m encountering an issue when adding/removing a service account from a pod spec (in a Deployment):

Spec: &corev1.PodSpecArgs{
					ServiceAccountName: tenant.serviceAccount.Metadata.Name(),

Let’s say I had the above, and then I just deleted the ServiceAccountName line. Pulumi seems to recognize that it should do an update, but nothing happens on the Kubernetes side (the Deployment is not updated, the pod does not restart), and Pulumi hangs waiting for something to happen (

[diff: ~spec]; [1/2] Waiting for app ReplicaSet be marked available (1/1 Pods available)

). Any ideas what might be going on?

Has anyone encountered this in the CI pipelines (AzDO) recently? Nothing else changed other than pipelines are failing now. Running the pulumi apps from the console still works, so we are doing this manually now.

configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Question that spans Kubernetes and AWS: I have an existing EKS cluster I do not wish to import, but I would like to get the OIDC info for it. I’ve successfully been able to get the URL like this:

eksCluster, err := eks.LookupCluster(ctx, &eks.LookupClusterArgs{
			Name: "my-eks-cluster",
		})
		if err != nil {
			return err
		}
		ctx.Export("eks-oidc-url", pulumi.String(eksCluster.Identities[0].Oidcs[0].Issuer))

This is using the

aws/eks

package (not

pulumi-eks

). However,

Issuer

is just the URL string, and does not include the ARN. I wanted to do something like a lookup on the

iam.OpenIdConnectProvider

, but it doesn’t look like it is a supported operation (https://www.pulumi.com/docs/reference/pkg/aws/iam/openidconnectprovider), as I am only able to lookup an existing resource, but this is already pre-created. Is there a way to do this, or do I have to use the AWS SDK for this?

I’m having some challenges in understanding

ProviderCredentialOpts

. For example, I have a Pulumi stack in which I set (via config) the AWS access token and secret, as it is in a different account than the one my AWS CLI (i.e.

~/.aws

) is configured with. I then create a cluster like this:

cluster, err := eks.NewCluster(ctx, "my-cluster", &eks.ClusterArgs{
		VpcId:            vpc.ID(),
		PublicSubnetIds:  pulumi.ToStringArrayOutput(publicSubnetIDs),
		PrivateSubnetIds: pulumi.ToStringArrayOutput(privateSubnetIDs),
		EnabledClusterLogTypes: pulumi.StringArray{
			pulumi.String("api"),
			pulumi.String("audit"),
			pulumi.String("authenticator"),
		},
		SkipDefaultNodeGroup: pulumi.BoolPtr(true),
		InstanceRoles: iam.RoleArray{
			// role0,
			// role1,
			role2,
		},
		NodeAssociatePublicIpAddress: pulumi.Bool(false),
		Version:                      pulumi.String("1.20"),
		UseDefaultVpcCni:             pulumi.Bool(true),
	})
	if err != nil {
		return err
	}

This works fine, but I get an error at the end basically saying that it could not connect to the cluster (I believe to do the CNI or other settings) as it could not authenticate. In order to enable authentication, I had to: 1. Create a new profile in my

~/.aws

folder that had the credentials set for this new account (I called it

ssa

) 2. Add the following to the above cluster create:

ProviderCredentialOpts: eks.KubeconfigOptionsArgs{
			ProfileName: pulumi.String("ssa"),
		},

Now it could connect properly/errors were gone. However, I am not quite sure I am following why this is necessary, and the docs/examples are a bit sparse. Specifically, I am a bit concerned that I have to specify a specific profile to use (one that I have to configure out of band on whatever machine is running Pulumi), which doesn’t seem easily repeatable. Given Pulumi already has the AWS credentials to use to authenticate to create the cluster, why can’t it use those when talking to Kubernetes proper?

I created a EKS cluster in one project (say infra/dev) and try to use it from another project (say app/dev) using stackReference. I can retrieve kubeconfig which is exported in the infra/dev stack. I can create a k8sprovider with it, I can then use the k8sprovider to create additional resource, for example, namespace/serviceaccount for the app/dev. But how do I get the EKS cluster itself from it? I need the EKS cluster to retrieve OidcProvider.

Hey all! I have an ingress that is managed via the aws LB controller. I use its tls cert as the secretName in the tls section of the ingress, and list the three hosts that are in the rules section of the ing.

aws-load-balancer-tls

I use a transformation in the LB controller to tell Pulumi to ignoreChanges on the LB ctrl tls secret. Subsequent ups regenerate the secret causing problems with the ing’s. I also make all ing’s dependsOn the cntrl. This is bec the cntrl pods don’t spin up fast enough, and the ing is still attempted to be created. I get a service not found error. So it is a race condition. This is the current context of my setup. I want to try auto tls discovery via host used in ing rules. I tried removing tls section but it wouldn’t find certs. Even though I have certs created in ACM. Perhaps it can’t find ACM certs? the error I get:

Failed build model due to ingress: waterrecharge/waterrecharge-ingress: none certificate found for host: <http://waterrecharge-pulumi-api-aryeh.tqhosted.com|waterrecharge-pulumi-api-aryeh.tqhosted.com>"

That’s referring to the ACM cert i’ve created, right? It can’t find it? I try putting back the tls section in the ing, and I still have the same issue. What’s happening is bec of that error, the ing doesn’t resolve in pulumi. I can’t get the hostname. So the record alias can’t be created. I get:

error: aws:route53/record:Record resource '<http://waterrecharge-pulumi-aryeh.tqhosted.com|waterrecharge-pulumi-aryeh.tqhosted.com>' has a problem: Required attribute is not set. Examine values at 'Record.Aliases'.

I have to do a refresh for pulumi to get the ing data and be able to resolve the hostname.

any ideas what is going on?

To summarize this, using .networking ingress, we get this error:

Failed build model due to ingress: waterrecharge/waterrecharge-ingress: none certificate found for host: <http://waterrecharge-pulumi-api-aryeh.tqhosted.com|waterrecharge-pulumi-api-aryeh.tqhosted.com>"

With the release of Pulumi 3.4.0 (which should include https://github.com/pulumi/pulumi/pull/7158), any chance we can get

pulumi-eks

updated as well to handle https://github.com/pulumi/pulumi-eks/issues/566 and https://github.com/pulumi/pulumi-eks/issues/577?

Anyone have any good thoughts on the best way to get istio installed and configured via pulumi at the moment? I feel like istio have doubled down on istioctl, at the same time the helm chart situation isn't exactly optimal..

Hi, Is there a way to update a kubernetes secret into a pulumi program? The secret is generated by a helm chart that is installed by pulumi and the program has to change its secret. It’s actually for ArgoCD.

What’s the best way to resolve this type of issue:

error: pre-step event returned an error: failed to verify snapshot: resource urn:pulumi:ssa-us-west-2::okera-infra-regions::kubernetes:yaml:ConfigFile$kubernetes:core/v1:ServiceAccount::cert-manager/cert-manager-webhook refers to unknown provider urn:pulumi:ssa-us-west-2::okera-infra-regions::pulumi:providers:kubernetes::k8s-ssa-provider::460da6b8-808b-4d03-b8f8-ee2fdc9ec693

I get this during

pulumi up -f

, but same issue with if I do

pulumi refresh

I’ve deleted the EKS cluster that a bunch of my resources were deployed onto and created a new one (this is all with Pulumi in a different project/stack). In the stack that was in charge of those resources, I am trying to do a

pulumi up

, and it created resources on my new cluster just fine, but is failing to delete the old resources since that EKS cluster no longer exists. 1. Is this expected? 2. How should I get out of this situation? The old resources can’t really exist by definition anymore since the EKS cluster is gone.

pulumi refresh

errors out in the same way.

error: no resource plugin 'kubernetes-v1.1.1' found in the workspace or on your $PATH, install the plugin using `pulumi plugin install resource kubernetes v1.1.1`

and by doing so, I get

error: [resource plugin kubernetes-1.1.1] downloading from : 403 HTTP error fetching plugin from <https://get.pulumi.com/releases/plugins/pulumi-resource-kubernetes-v1.1.1-darwin-amd64.tar.gz>

Has anyone faced an issue where the Helm chart for cert-manager doesn’t deploy the Deployment objects on the first try? (Using AKS, Python, latest pulumi & pulumi-kubernetes) - Or more broadly the case where a Helm chart is not deployed in it’s entirety?

+   │        ├─ kubernetes:<http://helm.sh/v3:Chart|helm.sh/v3:Chart>                                                   iaaksuksouthmm13005-cert-manager                                                     created                 
 +   │        │  ├─ kubernetes:core/v1:ServiceAccount                                          nginx-ingress/iaaksuksouthmm13005-cert-manager-cainjector                            created                 
 +   │        │  ├─ kubernetes:core/v1:ServiceAccount                                          nginx-ingress/iaaksuksouthmm13005-cert-manager                                       created                 
 +   │        │  ├─ kubernetes:core/v1:Service                                                 nginx-ingress/iaaksuksouthmm13005-cert-manager                                       **creating failed**     1 error
 +   │        │  ├─ kubernetes:core/v1:ServiceAccount                                          nginx-ingress/iaaksuksouthmm13005-cert-manager-webhook                               created                 
 +   │        │  ├─ kubernetes:core/v1:Service                                                 nginx-ingress/iaaksuksouthmm13005-cert-manager-webhook                               **creating failed**     1 error
 +   │        │  ├─ kubernetes:<http://admissionregistration.k8s.io/v1:MutatingWebhookConfiguration|admissionregistration.k8s.io/v1:MutatingWebhookConfiguration>    iaaksuksouthmm13005-cert-manager-webhook                                             created                 
 +   │        │  ├─ kubernetes:<http://admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration|admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration>  iaaksuksouthmm13005-cert-manager-webhook                                             created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-cainjector                                          created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-issuers                                  created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-clusterissuers                           created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-certificates                             created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-orders                                   created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-challenges                               created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-ingress-shim                             created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-view                                                created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-edit                                                created                 
 +   │        │  ├─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-controller-approve:cert-manager-io                  created                 
 +   │        │  └─ kubernetes:<http://rbac.authorization.k8s.io/v1:ClusterRole|rbac.authorization.k8s.io/v1:ClusterRole>                        iaaksuksouthmm13005-cert-manager-webhook:subjectaccessreviews                        created

Those services fail to deploy because it hasn’t deployed the deployments. If I re-run, it will then attempt to do the deployments

How to disable ambient k8s? re: https://www.pulumi.com/docs/intro/cloud-providers/kubernetes/setup/#kubernetes-configuration I'd like to do a

config.require('kubernetes:context')

to ensure only the expected cluster gets modified by Pulumi, but this seems to be scoped to the stack I want to make sure folks who fork my pulumi program don't mess up their default k8s context, whatever that happens to be on their machines... I'd like them to be explicit about the k8s cluster they want to target

Will deleting a stack cascade-delete dependent stacks? Re: https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences It seems it might in the example in the docs, but what if the output isn't a k8s provider kubeconfig? Say, I output an IP address that I use in a dependent stack as configuration for another resource, won't this corrupt my dependent stack's state?

I'm installing the aws load balancer controller via the helm chart. Every time I run update it wants to create-replacement the tls secret.

How do I set managed node group tags for the underlying cluster at the cluster creation time? I'd like to set these 2 tags so cluster autoscaler can auto-discover the node group.

<http://k8s.io/cluster-autoscaler/|k8s.io/cluster-autoscaler/><cluster-name>

owned

<http://k8s.io/cluster-autoscaler/enabledTRUE|k8s.io/cluster-autoscaler/enabledTRUE>

How to access a service account's secret programmatically? Goal is to output the base64 decoded secret as an output variable, and refactor this hack:

export const getTokenWith = pulumi.interpolate `kubectl get secret/${dashboardServiceAccount.secrets[0].name} -n kube-system -o go-template='{{.data.token | base64decode}}'`

which "works" by returning:

getTokenWith: "kubectl get secret/admin-user-ys16knlv-token-xpqss -n kube-system -o go-template='{{.data.token | base64decode}}'"

which I then copy paste to get what I really need to log into the kubernetes dashboard with a token...

Has anyone done Volume Snapshots via Pulumi yet? https://kubernetes.io/docs/concepts/storage/volume-snapshots/

this is related to the following peculiar challenge: I have some statefulsets that need, before they first start, have their volume seeded with 3-400GB of data. (all stateful sets the same data). The best I could do atm is upload this data to S3 and have an init container that uses the aws cli to download this data. But this takes quite some time on start. The data needs to be in the volume as it is a “snapshot” that needs read/write when the main container starts and cannot be write many. So i’m looking to make this blazing fast.

@gorgeous-egg-16927 Hey Levi... Not sure whom else to ask.. but we run into this scenario a lot with our development clusters and was wondering how you felt or what you thought. We use pulumi in separate apps to: 1. Stand up the cluster 2. Install core k8s resources (fluentd, certmanager, nginx/traefik, etc) 3. Install LOB apps Since the clusters are for lower environments we tear them down occassionally. The challenge with this is unless we tear out all of the installed k8s resources first, the pulumi state for all of the k8s resources is never cleared out and we have to go do that manually. Which we understand but it still happens on occasion. Is there a feature or technique present in pulumi where if a "parent stack" is deleted, all child stacks state are automatically cleared out as well?

Hey guys. Currently using the latest versions of pulumi (python sdk) and pulumi-kubernetes, and seeing this: File “/pulumi/projects/venv/lib/python3.9/site-packages/pulumi_kubernetes/apiextensions/v1/CustomResourceDefinition.py”, line 121, in init self._internal_init(resource_name, *args, **kwargs) TypeError: _internal_init() got an unexpected keyword argument ‘status’ I’ve considered modifying the upstream yaml files to remove the offending code, but I’d prefer to either wait for the bug to be resolved, or roll back to an earlier version of pulumi - I believe v2 did not have this issue. Could anyone tell me if they have seen this issue and, if so, how they went about resolving? Alternatively, could someone offer up their requirements.txt file with a list of compatible python library versions? Thanks!