pulumi-community #kubernetes

red-lighter-44012

02/18/2021, 8:02 AM

I've been working on setting up one-click cluster provisioning automation so I can spin up clusters on demand (be it for test environments or disaster recovery). So far I'm pretty happy with everything but one thing - the ingress controller(s). - Do you install the ingress controller as a part of your initial cluster deployment? E.g. pulumi up installs some helm charts for the controller and thats it? - How do you handle ingress updates? This sounds like a potentially dangerous operation that should ideally be tested manually - Upgrading the ingress might work but .. - Installing the target version on a fresh cluster might fail or not work as expected - and especially vice versa: new installs are good, but helm upgrade breaks something (CRDs can be a pain) Currently I have a manual step that I do for cluster initialization. pulumi output (ingress IP, namespace name etc) is passed to the helm chart script that performs the initial ingress installation and this works pretty great. The problem is, that helm is not declarative and that on upgrade there are sometimes breaking changes (eg CRDs have to be manually applied). E.g. end up having to do perform these manual steps to ensure there's no downtime. It's literally easier to spin up a new cluster and gradually switch traffic over tbh.

proud-pizza-80589

02/19/2021, 7:43 AM

I’m using fetchOpts.repo to install helm charts out of custom repo’s, but i now have to install https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus which needs 2 repo’s so it can install dependencies. Any idea how to configure pulumi for these two repo’s?

colossal-australia-65039

02/20/2021, 2:55 AM

I'm following this blog for EKS cluster autoscaling (with managed node groups) and it's not working. My nodes stay at 1 even though there are unscheduled pods. When comparing it to the steps AWS gives, the Pulumi blog is skipping over a bunch of resources to provision. Not sure if this blog is no longer applicable since it was published in 2019, or if there's something that I'm missing.

bumpy-laptop-30846

02/24/2021, 1:53 PM

Hi guys, I am creating a cluster with fargate on aws, with awsx and eks libraries. The issue is that my deployment is not deploying on fargate. I think it might be due to the namespace of fargate. If one create the cluster like this:

Copy code

const cluster = new eks.Cluster( clusterName,
  {
    fargate: true,
    vpcId: vpc.id,
    publicSubnetIds: vpc.publicSubnetIds,
    privateSubnetIds: vpc.privateSubnetIds,
    deployDashboard: false,
    nodeAssociatePublicIpAddress: false,
    providerCredentialOpts: {},
    skipDefaultNodeGroup: true, ...

then fargate use the default namespace? Is there a way to access the fargateProfile when creating the fargate profile with

fargate: true

lemon-monkey-228

02/26/2021, 12:12 PM

Is there a way to force pulumi to always render resources from fresh instead of diffs

lemon-monkey-228

02/26/2021, 12:30 PM

I’ve resorted to piping the stack.json through

jq

and deleting the resources key

lemon-monkey-228

02/26/2021, 12:31 PM

to ensure it’s rendered from fresh when required

orange-psychiatrist-22511

02/26/2021, 2:12 PM

quick question: is there a benefit to creating resources with random suffixes (the default behavior with pulumi)?

breezy-cricket-40277

02/26/2021, 3:50 PM

Is there any way to quickly template helm with a

dry-run

before applying it? I have a helm chart deployed and don’t want to deploy again without

dry-running

the helm template first.

wet-noon-14291

02/26/2021, 10:30 PM

Has anyone an example of creation of a service account and then also generation of the kubeconfig for that service account? I know I can probably do it on the command line, but I would like to do it in pulumi but can't find the right methods to do it. Basically this but in pulumi: https://docs.armory.io/docs/armory-admin/manual-service-account/

dry-engine-17210

02/27/2021, 6:32 PM

Anyone tried to build a GKE Autopilot cluster using Pulumi yet? Any gotchas?

👀 1

bitter-application-91815

03/03/2021, 4:53 PM

hey folks, when creating a eks cluster how do i ensure that the cluster is accessible from both public and private

bitter-application-91815

03/03/2021, 4:53 PM

adorable-action-51248

03/05/2021, 9:31 AM

Hi ! I am wondering what is the best way to setup K8S with GKE and control the subnet such that the cluster is “VPC-native” and has the maximum number of pods per node (110). Currently, I have a setup where i use

gcp.compute.Subnetwork

with empty

secondaryIpRanges: []

and then refer to that subnet in

gcp.container.Cluster

and use

ipAllocationPolicy

to specify the services and pods subnets. However, this setup breaks because GKE modifies the network’s

secondaryIpRanges

and on the next

pulumi up

pulumi tries to remove GKEs modifications. The fails because the subnetwork is already in use.

colossal-australia-65039

03/05/2021, 11:39 PM

I'm using the

@pulumi/eks

Node package and when creating a

new Cluster()

I can supply

roleMappings

. Ideally these roleMappings would reference

ClusterRoleBindings

but these

ClusterRoleBindings

cannot exist until the EKS cluster itself is created since they're k8s resources. I'm stuck in a chicken/egg scenario for the initial creation of the EKS cluster! Is there a way to solve this without having to fall. back to referencing a hardcoded string value instead of my ideal case of referencing the

clusterRoleBinding.subjects[0].name

limited-planet-95090

03/08/2021, 11:52 PM

Hey folks, I’m working on adding code examples to get to language parity across our top five code examples in AWS, Azure, Google Cloud, and Kubernetes. For Kubernetes, these are the tutorials I’m adding examples for: EKS - Migrating Node Groups with Zero Downtime (https://www.pulumi.com/docs/tutorials/kubernetes/eks-migrate-nodegroups/) Deploy Nginx to a Kubernetes Cluster (https://www.pulumi.com/docs/tutorials/kubernetes/exposed-deployment/) Kubernetes Helm Wordpress Chart (https://www.pulumi.com/docs/tutorials/kubernetes/wordpress-chart/) Guestbook App with Redis and Nginx(https://www.pulumi.com/docs/tutorials/kubernetes/guestbook/) Graceful App Rollout (https://www.pulumi.com/docs/tutorials/kubernetes/configmap-rollout/) Do these feel like the most important examples you’ve used or are there other examples from our examples repo that have been more helpful? Are there different examples you’d like to see? Thanks, Dave

bitter-application-91815

03/10/2021, 7:07 PM

Hey guys, are there any examples of how to roll out the cluster-autoscaler helm chart. It fails to talk to the pods with

bitter-application-91815

03/10/2021, 7:08 PM

Copy code

the Kubernetes API server reported that "default/asg-chart-cloud-production-aws-cluster-autoscaler" failed to fully initialize or become live: 'asg-chart-cloud-production-aws-cluster-autoscaler' timed out waiting to be Ready
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

sticky-match-71841

03/11/2021, 1:52 PM

Can't the skipAwait annotations be used to ignore the status of a deployment? Seems to not work the way we expect it to 🤔 Some context: We would like to just configure a deployment but ignore if it ends up crash looping. From our PoV the deployment is configured correctly, so we don't care if it starts crash looping. Is it maybe a possibility to explicitly ignore the status?

limited-rainbow-51650

03/11/2021, 1:53 PM

I’m missing Pulumi datasources in the kubernetes provider. Is there any specific reason why this provider doesn’t have any

get

functions? E.g.

getService

that I can feed some labels or annotations?

adorable-action-51248

03/11/2021, 4:04 PM

Is there a good way to wait for services to be ready when they are deployed using

k8s.yaml.ConfigFile

? I tried using dependsOn but that doesnt wait long enough.

some-elephant-30417

03/12/2021, 8:29 AM

Hi! I am trying to do this to fetch a chart from https://helm.dask.org/.

Copy code

dask = helm.v3.Chart(
        f'dask-helm-{resource_suffix}',
        config=k8s.helm.v3.ChartOpts(
            chart='dask',
            repo='dask',
            version='4.5.7',
            fetch_opts=k8s.helm.v3.FetchOpts(
                repo='<https://helm.dask.org>',
            ),
        ),
        opts=pulumi.ResourceOptions(
            providers={ 'kubernetes': k8s_provider },
        ),
    )

But I get this error:

Copy code

error: Program failed with an unhandled exception:
    error: Traceback (most recent call last):
      File "/home/alexandre/.pyenv/versions/daks/lib/python3.9/site-packages/pulumi/runtime/invoke.py", line 110, in do_invoke
        return monitor.Invoke(req)
      File "/home/alexandre/.pyenv/versions/daks/lib/python3.9/site-packages/grpc/_channel.py", line 923, in __call__
        return _end_unary_response_blocking(state, call, False, None)
      File "/home/alexandre/.pyenv/versions/daks/lib/python3.9/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
        raise _InactiveRpcError(state)
    grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    	status = StatusCode.UNKNOWN
    	details = "invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to pull chart: chart "dask/dask" version "4.5.7" not found in <https://helm.dask.org> repository"
    	debug_error_string = "{"created":"@1615537561.526221208","description":"Error received from peer ipv4:127.0.0.1:38703","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to pull chart: chart "dask/dask" version "4.5.7" not found in <https://helm.dask.org> repository","grpc_status":2}"

I tried many variations of the

Chart

parameters without success. Any idea? Thank you!

brash-house-42711

03/15/2021, 10:00 PM

Hi. I'm trying to add an annotation on a service account. The code seems to work fine, at least I can see that the annotation gets added by

setIamRoleArn

method but Pulumi doesn't detect any changes. I wonder if annotations attribute is being ignores as there is a high change of it being updated outside of Pulumi? If so is there a way to force adding specific annotation? Thank you!

Copy code

private deployCloudWatchAgentDaemonset(): k8s.yaml.ConfigFile {
    let serviceAccounts = this.serviceAccounts;

    return new k8s.yaml.ConfigFile('cloudwatch-agent-setup', {
        file: ContainerInsights.CW_AGENT_TEMPLATE,
        transformations: [(obj: any, _opts: pulumi.CustomResourceOptions) => {
            if (typeof serviceAccounts !== 'undefined') {
                ContainerInsights.setIamRoleArn(obj, serviceAccounts);
            }
        }],
    },
    { providers: { kubernetes: this.k8sProvider } });
}

private static setIamRoleArn(obj: any, serviceAccounts: pulumi.Output<any>): void {
    if (obj !== undefined && obj.kind == 'ServiceAccount') {
        serviceAccounts.apply(serviceAccounts => {
            if (typeof serviceAccounts !== 'undefined' && Object.keys(serviceAccounts).includes(obj.metadata.name)) {
                if (!obj.metadata.annotations) {
                    obj.metadata['annotations'] = {}
                }
                obj.metadata.annotations['<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>'] = serviceAccounts[obj.metadata.name].role.arn;
            }
        });
    }
}

hundreds-battery-67030

03/16/2021, 12:49 AM

Is there any correlation between the memory taken up by

pulumi up

process and the number of objects in the K8s cluster? I have a situation where

pulumi up

was crawling to halt on a 8GB worker in CircleCI, and when I tried it locally I saw it taking up 20+GB of memory. Any tips on troubleshooting this further?

incalculable-animal-125

03/16/2021, 9:55 AM

Hello! How to do something like:

kubectl --kubeconfig ${PATH_TO_KUBECONFIG} --cluster ${CLUSTER_NAME} --token ${TOKEN}

?I'm confused on how to set the

--token

part with Pulumi. From here, seems like it is

id_token

https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

prehistoric-coat-10166

03/16/2021, 2:30 PM

Hi! Not sure if this is the appropriate channel, but since it is related to the

pulumi-kubernetes

package I figured to try here first. I'm trying to write some unit tests in C# for my stacks and I'm having trouble with a stack which contains a

Helm.V3.Chart

. The problem can be produced by having the following stack:

Copy code

public class FailingHelmStack : Pulumi.Stack
{
    public FailingHelmStack()
    {
        var chart = new Pulumi.Kubernetes.Helm.V3.Chart("chart"
            , new Pulumi.Kubernetes.Helm.ChartArgs()
            {
                Chart = "ingress-nginx",
                Namespace = "kube-system",
                FetchOptions = new Pulumi.Kubernetes.Helm.ChartFetchArgs
                {
                    Repo = "<https://kubernetes.github.io/ingress-nginx>",
                },
            });
    }
}

Using

Deployment.TestAsync

with this stack results in the following exception being thrown.

Copy code

Pulumi.RunException : Running program '<path>\bin\Debug\net5.0\testhost.dll' failed with an unhandled exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Collections.Immutable.ImmutableArray.CreateRange[TSource,TResult](ImmutableArray`1 items, Func`2 selector)
   at Pulumi.Extensions.SelectAsArray[TItem,TResult](ImmutableArray`1 items, Func`2 map)
   at Pulumi.InputList`1.op_Implicit(ImmutableArray`1 values)
   at Pulumi.Kubernetes.Helm.V3.Chart.<>c__DisplayClass3_0.<ParseTemplate>b__0(ImmutableArray`1 objs)
   at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
   at Pulumi.Output`1.ApplyHelperAsync[U](Task`1 dataTask, Func`2 func)
   at Pulumi.Output`1.Pulumi.IOutput.GetDataAsync()
   at Pulumi.Serialization.Serializer.SerializeAsync(String ctx, Object prop, Boolean keepResources)
   at Pulumi.Deployment.SerializeFilteredPropertiesAsync(String label, IDictionary`2 args, Predicate`1 acceptKey, Boolean keepResources)
   at Pulumi.Deployment.SerializeAllPropertiesAsync(String label, IDictionary`2 args, Boolean keepResources)
   at Pulumi.Deployment.RegisterResourceOutputsAsync(Resource resource, Output`1 outputs)
   at Pulumi.Deployment.Runner.<>c__DisplayClass9_0.<<WhileRunningAsync>g__HandleCompletion|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Pulumi.Deployment.Runner.WhileRunningAsync()
    Stack Trace:
       at Pulumi.Deployment.TestAsync(IMocks mocks, Func`2 runAsync, TestOptions options)
    <snip>
--- End of stack trace from previous location ---

I think I might be able to workaround this problem by mocking some necessary outputs perhaps. But I'm having trouble figuring out what exactly is required. Any help or suggestions would be greatly appreciated.

purple-plumber-90981

03/16/2021, 11:34 PM

hi room….python/eks question… im trying to create an EKS cluster with a nodegroup that uses spot instances. in trying to create my nodegroup :-

Copy code

eks_nodegroup = eks.NodeGroup("my-eks-nodegroup", opts=eks_opts, **eks_node_group_config)

File "/Users/bmeehan/repos/itplat-pulumi-infrastructure/venv/lib/python3.7/site-packages/pulumi_eks/node_group.py", line 145, in __init__
        raise TypeError("Missing required property 'cluster'")
    TypeError: Missing required property 'cluster'

however in trying to create my cluster it seems to need nodegroup to pre-exist :-

Copy code

# node_group_options: Optional[pulumi.Input[pulumi.InputType['ClusterNodeGroupOptionsArgs']]] = None,
so in my eks_cluster_config
"node_group_options": eks_node_group_config,
eks_cluster = eks.Cluster("myt-eks-cluster", opts=eks_opts, **eks_cluster_config)

so which should come first? chicken or egg ?

adorable-action-51248

03/17/2021, 9:24 AM

Hi ! i have multiple stacks that are using the same k8s namespace. is there a good way to do a “get or create” namespace ? or do i to need create a stack with the namespace and then pull it in with “requireOutput” in all the other stacks ?

delightful-mouse-18472

03/17/2021, 11:31 AM

Hi ppl, I'm using a Helm Chart to deploy part of my stack and in the Chart there's a Secrete I update in an init Container. Is there a way to get the Secret with its updated data after creating the Chart instance (I'm using Typescript)?

proud-pizza-80589

03/18/2021, 1:54 PM

I have a stacks, one with 3 clusters exporting their kubeconfig like

providerregionkubeconfig

. A second stack which uses a stack reference and the predicable name to fetch the output, and create a provider to deploy stuff on the cluster. Since we are messing about with the clusters sometimes they need to be recreated. At that point the second stack is broken, i cannot destroy nor up since the cluster is replaced (error:

configured Kubernetes cluster is unreachable: unable to load schema information from the API server

) The fix is to manually edit the stack export json files are remove all references to stuff deployed on the cluster so i can destroy it. I’m looking for a) a way to not depend on the specific k8s api server endpoint but on what it gets from the stack reference or b) a --ignore-errors and delete whatever you can find option removing the things you cannot. Any hints?