https://pulumi.com logo
Join Slack
Powered by
Hi folks, i’m seeking to understand a pulumi behav...
# kubernetes
p
Hi folks, i’m seeking to understand a pulumi behaviour. I deploy both the “aws efs csi driver” and “aws load balancer controller” via helm (pulumi wrapped helm) and every time i “pulumi up” these resources get replaced/updated regardless of the fact the code is unchanged…. all thoughts and learning welcome, code snippets in thread
Copy code
k8s_provider = k8s.Provider(
    "k8s_provider",
    context=current_context,
    kubeconfig='~/.kube/config',
)

# setup EFS CSI driver via helm
k8s_h_efscsi = k8s.helm.v3.Chart(
    "itplat-helm-efs-driver",
    k8s.helm.v3.ChartOpts(
        chart="aws-efs-csi-driver",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://kubernetes-sigs.github.io/aws-efs-csi-driver/>",
        ),
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)

# setup aws-load-balancer-controller (ALB) via helm
k8s_h_alb_controller = k8s.helm.v3.Chart(
    "itplat-helm-alb-controller",
    k8s.helm.v3.ChartOpts(
        chart="aws-load-balancer-controller",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://aws.github.io/eks-charts>",
        ),
        namespace='kube-system',
        transformations=[remove_status],
        values={"clusterName": pulumi_stack_info['name']},
        # values={"clusterName": "ipd-eks-use1"},
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)
so i can 
pulumi up
multiple times with no code change and every time the resources are replaced/updated
when i look at the details of the change it is always the contents of ca/cert values being changed
Copy code
~ kubernetes:<http://admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration|admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration>: (update)
            [id=aws-load-balancer-webhook]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::aws-load-balancer-webhook|helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::aws-load-balancer-webhook>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ webhooks: [
              ~ [0]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"


              ~ [1]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"


        ~ kubernetes:<http://admissionregistration.k8s.io/v1:MutatingWebhookConfiguration|admissionregistration.k8s.io/v1:MutatingWebhookConfiguration>: (update)
            [id=aws-load-balancer-webhook]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration::aws-load-balancer-webhook|helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration::aws-load-balancer-webhook>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ webhooks: [
              ~ [0]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"

              ~ [1]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"

        --kubernetes:core/v1:Secret: (delete-replaced)
            [id=kube-system/aws-load-balancer-tls]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
        +-kubernetes:core/v1:Secret: (replace)
            [id=kube-system/aws-load-balancer-tls]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ data: {
              ~ ca.crt : "stuff"

              ~ tls.crt: "stuff"

              ~ tls.key: "stuff"


        ++kubernetes:core/v1:Secret: (create-replacement)
            [id=kube-system/aws-load-balancer-tls]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ data: {
              ~ ca.crt : "stuff"

              ~ tls.crt: "stuff"

              ~ tls.key: "stuff"
b
@purple-plumber-90981 seems like you are good?
p
just tryin to work out if the “cert manager” they talk about (in the thread above) is ACM or an in-k8s resource
and not sure how this relates to the EFS CSI driver part
looks like its a kubernetes resource - 
<https://github.com/aws/eks-charts/issues/419>
ie 
<https://github.com/jetstack/cert-manager>
all the learnings today 😉
trying
Copy code
# setup jetstack cert-manager via helm
k8s_h_cert_manager = k8s.helm.v3.Chart(
    "itplat-helm-cert-manager",
    k8s.helm.v3.ChartOpts(
        chart="cert-manager",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://charts.jetstack.io>",
        ),
        namespace='kube-system',
        values={
            "clusterName": pulumi_stack_info['name'],
            "installCRDs": "true",
        },
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)
TypeError: _internal_init() got an unexpected keyword argument ‘status’
i think i have seen this before where the helm chart includes encoding of the output… there is a way to bypass it, just otta dig it out of my history
found it :-
Copy code
# for <https://github.com/pulumi/pulumi-kubernetes/issues/800>
# where an upstream helm chart invalidly defines a key for the output only resource "status"
def remove_status(obj):
    if obj["kind"] == "CustomResourceDefinition" and "status" in obj:
        try:
            del obj["status"]
        except KeyError:
            pass
w
@purple-plumber-90981 Yes, I'm installing jetstack's cert manager and leveraging that where possible in other charts.
p
yeah i figured it out eventually . . thx… im just trying to give details in my threads for those who follow after and have the same prooblems
w
Beware that there is a race condition with cert manager being ready and subsequently issuing certs
p
a lot of threads in here just have “yeah its fixed now” without any details, which doesnt really help others
w
I handle that by issuing a test cert and depending on that. Happy share code (in dotnet/c#).
Copy code
// cert manager; <https://github.com/jetstack/cert-manager/tree/master/deploy/charts/cert-manager>
Logger.LogDebug("Installing cert manager");
var certManagerCrds = new ConfigGroup("cert-manager-crds",
    new ConfigGroupArgs { Yaml = ReadResource("CertManagerCrds.yaml") },
    new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

var certManagerNs = new Namespace("cert-manager",
    new NamespaceArgs { Metadata = new ObjectMetaArgs { Name = "cert-manager" } },
    new CustomResourceOptions { Provider = k8sProvider });

var certManagerValues =
    new Dictionary<string, object>
    {
        ["prometheus"] = new
        {
            enabled = true,
            servicemonitor = new { enabled = true }
        }
    };

var certManagerChart = new Chart("cert-manager",
    new ChartArgs
    {
        Namespace = "cert-manager",
        FetchOptions = new ChartFetchArgs { Repo = "<https://charts.jetstack.io>" },
        Chart = "cert-manager",
        Version = K8sConfig.CertManagerChartVersion,
        Values = certManagerValues,
        SkipCRDRendering = true
    },
    new ComponentResourceOptions { DependsOn = { certManagerCrds, certManagerNs, kubePrometheusStackCrds }, Provider = k8sProvider });

var certManager = certManagerChart.GetResource<Deployment>("cert-manager", "cert-manager").AsResource();
var certManagerCaInjector = certManagerChart.GetResource<Deployment>("cert-manager-cainjector", "cert-manager").AsResource();
var certManagerWebHook = certManagerChart.GetResource<Deployment>("cert-manager-webhook", "cert-manager").AsResource();

var certManagerTest = new ConfigGroup("cert-manager-test",
    new ConfigGroupArgs { Yaml = RenderTemplate("CertManagerTest.yaml", ReadResource, new { @namespace = "cert-manager" }) },
    new ComponentResourceOptions { DependsOn = { certManager, certManagerCaInjector, certManagerWebHook }, Provider = k8sProvider });

var certManagerTestCert = certManagerTest.GetCustomResource("<http://cert-manager.io/v1/Certificate|cert-manager.io/v1/Certificate>", "cert-manager-test-cert", "cert-manager").AsResource();
//var certManagerTestIssuer = certManagerTest.GetCustomResource("<http://cert-manager.io/v1/Issuer|cert-manager.io/v1/Issuer>", "cert-manager-test-issuer", "cert-manager").AsResource();
p
so my final pulumi code deploy including the status fix looks like :-
Copy code
# for <https://github.com/pulumi/pulumi-kubernetes/issues/800>
# where an upstream helm chart invalidly defines a key for the output only resource "status"
def remove_status(obj):
    if obj["kind"] == "CustomResourceDefinition" and "status" in obj:
        try:
            del obj["status"]
        except KeyError:
            pass


# setup jetstack cert-manager via helm
k8s_h_cert_manager = k8s.helm.v3.Chart(
    "itplat-helm-cert-manager",
    k8s.helm.v3.ChartOpts(
        chart="cert-manager",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://charts.jetstack.io>",
        ),
        namespace='kube-system',
        transformations=[remove_status],
        values={
            "clusterName": pulumi_stack_info['name'],
            "installCRDs": "true",
        },
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)
w
Copy code
// aws load balancer controller; <https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller>
            Logger.LogDebug("Installing aws load balancer controller");
            var awsLbcRole = new RoleX($"{k8sPrefix}-aws-load-balancer-controller",
                new RoleXArgs
                {
                    AssumeRolePolicy = IamHelpers.AssumeRoleForServiceAccount(oidcArn, oidcUrl, "kube-system", "aws-load-balancer-controller", awsProvider),
                    InlinePolicies = { ["policy"] = ReadResource("AwsLoadBalancerPolicy.json") }
                },
                new ComponentResourceOptions { Provider = awsProvider });

            var awsLbcCrds = new ConfigGroup("aws-load-balancer-controller-crds",
                new ConfigGroupArgs { Yaml = ReadResource("AwsLoadBalancerCrds.yaml") },
                new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

            var awsLbcValues = Output.Tuple(clusterName, awsLbcRole.Arn).Apply(((string ClusterName, string RoleArn) tuple) =>
                new Dictionary<string, object>
                {
                    ["clusterName"] = tuple.ClusterName,
                    ["enableCertManager"] = true,
                    ["serviceAccount"] = new { annotations = new Dictionary<string, string> { ["<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"] = tuple.RoleArn } }
                });

            var awsLbcChart = new Chart("aws-load-balancer-controller", // ingress records with <http://alb.ingress.kubernetes.io|alb.ingress.kubernetes.io> annotations depend on chart finalizers
                new ChartArgs
                {
                    Namespace = "kube-system",
                    FetchOptions = new ChartFetchArgs { Repo = "<https://aws.github.io/eks-charts>" },
                    Chart = "aws-load-balancer-controller",
                    Version = K8sConfig.AwsLbcChartVersion,
                    Values = awsLbcValues,
                    SkipCRDRendering = true,
                    ApiVersions = { "<http://admissionregistration.k8s.io/v1|admissionregistration.k8s.io/v1>" }
                },
                new ComponentResourceOptions { DependsOn = { awsLbcCrds, certManagerTestCert }, Provider = k8sProvider });

            var awsLbc = awsLbcChart.GetResource<Deployment>("aws-load-balancer-controller", "kube-system").AsResource();
Note how aws lbc depends on the cert manager test cert
I also install all crds separately as that's the recommend best practice
p
yeah i did that in python like
Copy code
# setup CRD's for aws-load-balancer-controller CRD's
k8s_k_alb_crd = k8s.kustomize.Directory(
    "itplat-alb-crd",
    directory="<https://github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master>",
    transformations=[remove_status],
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)

# setup aws-load-balancer-controller (ALB) via helm
k8s_h_alb_controller = k8s.helm.v3.Chart(
    "itplat-helm-alb-controller",
    k8s.helm.v3.ChartOpts(
        chart="aws-load-balancer-controller",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://aws.github.io/eks-charts>",
        ),
        namespace='kube-system',
        transformations=[remove_status],
        values={
            "clusterName": pulumi_stack_info['name'],
            "enableCertManager": "true",
        },
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_h_cert_manager, k8s_k_alb_crd], provider=k8s_provider),
)
so explicit depends on
ah i see you depend on a created cert… i like it
w
Depending on the helm chart is not the same as depending on all chart child resources, hence you currently need to get specific child resources and depend on those.
e.g. cert manager test does:
Copy code
var certManagerTest = new ConfigGroup("cert-manager-test",
    new ConfigGroupArgs { Yaml = RenderTemplate("CertManagerTest.yaml", ReadResource, new { @namespace = "cert-manager" }) },
    new ComponentResourceOptions { DependsOn = { certManager, certManagerCaInjector, certManagerWebHook }, Provider = k8sProvider });
p
im going to use that, awesome, thanks
w
cert manager test:
Copy code
---
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Issuer
metadata:
  name: cert-manager-test-issuer
  namespace: {{ namespace }}
spec:
  selfSigned: {}
---
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Certificate
metadata:
  name: cert-manager-test-cert
  namespace: {{ namespace }}
spec:
  dnsNames:
    - <http://example.com|example.com>
  secretName: cert-manager-test
  issuerRef:
    name: cert-manager-test-issuer
p
sweet, much appreciated
w
I render that template replacing the namespace, as shown in the code above.
p
and in fact i can see i should do similar things in other places in my code where i helm-deploy stuff…
w
I'm also using irsa, which I recommend for least privilege best practice. Most aws resources will have irsa setup details, like aws load balancer. Basically, you create an iam role with the least required aws permissions, and associate it with a k8s service account with an annotation.
p
nice, i will check it out
he @worried-city-86458 i am having issues with cert-manager not creating the deployment… i ’m guessing it is because the service .spec.selector is dependent on “cert-manager.name” and “.Release.Name” . . . did you have to tweek something to have either of these passed in your helmchart for cert-manager ?
ie how you would specify the release to helm cli like 
helm install my-release --namespace cert-manager jetstack/cert-manager
i cant seem to get past
Copy code
kubernetes:core/v1:Service (cert-manager/itplat-helm-cert-manager):
    error: 2 errors occurred:
    	* resource cert-manager/itplat-helm-cert-manager was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'itplat-helm-cert-manager' timed out waiting to be Ready
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
w
p
only difference i see there is SkipCRDRendering = true and your dependence on a pre-existing prometheus stack
can you tell me in your cert-manager service, is you spec selector 
<http://app.kubernetes.io/name|app.kubernetes.io/name>
the same as the logical name for your chart ? (ie “cert-manager”)
im just trying to ensure what pulumi uses as the release name
oh wait no its spec.selector 
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>
that should be release name
w
p
much appreciated
r
@worried-city-86458 Hey... so I can't figure this out with the AzureNative SDK:
Copy code
var certManager = certManagerChart.GetResource<Deployment>("cert-manager", "cert-manager").AsResource();
var certManagerCaInjector = certManagerChart.GetResource<Deployment>("cert-manager-cainjector", "cert-manager").AsResource();
var certManagerWebHook = certManagerChart.GetResource<Deployment>("cert-manager-webhook", "cert-manager").AsResource();
What type is 
Deployment
and is 
AsResource()
an extension method from somewhere?
w
Pulumi.Kubernetes.Apps.V1.Deployment
and yes AsResource is an extension method to hide the ugly cast via apply
✅ 1
(Needed because subsequent 
DependsOn
is type 
InputList<Resource>
)
r
Ok great - thank you
w
hi, what was the solution to the original question? I can not access the posted solution 😞
( any time I'm executing 
pulumi up
, some of the resources of the alb ingress controller are being recreated 😞 )
4 Views
Open in Slack
PreviousNext
Powered by