https://pulumi.com logo
Powered by
Title
p

purple-plumber-90981

07/24/2021, 10:38 PM
Hi folks, i’m seeking to understand a pulumi behaviour. I deploy both the “aws efs csi driver” and “aws load balancer controller” via helm (pulumi wrapped helm) and every time i “pulumi up” these resources get replaced/updated regardless of the fact the code is unchanged…. all thoughts and learning welcome, code snippets in thread
k8s_provider = k8s.Provider(
    "k8s_provider",
    context=current_context,
    kubeconfig='~/.kube/config',
)

# setup EFS CSI driver via helm
k8s_h_efscsi = k8s.helm.v3.Chart(
    "itplat-helm-efs-driver",
    k8s.helm.v3.ChartOpts(
        chart="aws-efs-csi-driver",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://kubernetes-sigs.github.io/aws-efs-csi-driver/>",
        ),
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)

# setup aws-load-balancer-controller (ALB) via helm
k8s_h_alb_controller = k8s.helm.v3.Chart(
    "itplat-helm-alb-controller",
    k8s.helm.v3.ChartOpts(
        chart="aws-load-balancer-controller",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://aws.github.io/eks-charts>",
        ),
        namespace='kube-system',
        transformations=[remove_status],
        values={"clusterName": pulumi_stack_info['name']},
        # values={"clusterName": "ipd-eks-use1"},
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)
so i can 
pulumi up
multiple times with no code change and every time the resources are replaced/updated
when i look at the details of the change it is always the contents of ca/cert values being changed
~ kubernetes:<http://admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration|admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration>: (update)
            [id=aws-load-balancer-webhook]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::aws-load-balancer-webhook|helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration::aws-load-balancer-webhook>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ webhooks: [
              ~ [0]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"


              ~ [1]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"


        ~ kubernetes:<http://admissionregistration.k8s.io/v1:MutatingWebhookConfiguration|admissionregistration.k8s.io/v1:MutatingWebhookConfiguration>: (update)
            [id=aws-load-balancer-webhook]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration::aws-load-balancer-webhook|helm.sh/v3:Chart$kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration::aws-load-balancer-webhook>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ webhooks: [
              ~ [0]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"

              ~ [1]: {
                      ~ clientConfig: {
                          ~ caBundle: "stuff"

        --kubernetes:core/v1:Secret: (delete-replaced)
            [id=kube-system/aws-load-balancer-tls]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
        +-kubernetes:core/v1:Secret: (replace)
            [id=kube-system/aws-load-balancer-tls]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ data: {
              ~ ca.crt : "stuff"

              ~ tls.crt: "stuff"

              ~ tls.key: "stuff"


        ++kubernetes:core/v1:Secret: (create-replacement)
            [id=kube-system/aws-load-balancer-tls]
            [urn=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
            [provider=urn:pulumi:ipd-eks-use1-configure::aws_eks_create::pulumi:providers:kubernetes::k8s_provider::4618acaf-d8ba-417f-913f-a443426d5a46]
          ~ data: {
              ~ ca.crt : "stuff"

              ~ tls.crt: "stuff"

              ~ tls.key: "stuff"
aaaaand then i search back to find https://pulumi-community.slack.com/archives/CRFURDVQB/p1623821678245000
b

bored-table-20691

07/24/2021, 11:30 PM
@purple-plumber-90981 seems like you are good?
p

purple-plumber-90981

07/24/2021, 11:31 PM
just tryin to work out if the “cert manager” they talk about (in the thread above) is ACM or an in-k8s resource
and not sure how this relates to the EFS CSI driver part
looks like its a kubernetes resource - 
<https://github.com/aws/eks-charts/issues/419>
ie 
<https://github.com/jetstack/cert-manager>
all the learnings today 😉
trying 
# setup jetstack cert-manager via helm
k8s_h_cert_manager = k8s.helm.v3.Chart(
    "itplat-helm-cert-manager",
    k8s.helm.v3.ChartOpts(
        chart="cert-manager",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://charts.jetstack.io>",
        ),
        namespace='kube-system',
        values={
            "clusterName": pulumi_stack_info['name'],
            "installCRDs": "true",
        },
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)
TypeError: _internal_init() got an unexpected keyword argument ‘status’
i think i have seen this before where the helm chart includes encoding of the output… there is a way to bypass it, just otta dig it out of my history
found it :- 
# for <https://github.com/pulumi/pulumi-kubernetes/issues/800>
# where an upstream helm chart invalidly defines a key for the output only resource "status"
def remove_status(obj):
    if obj["kind"] == "CustomResourceDefinition" and "status" in obj:
        try:
            del obj["status"]
        except KeyError:
            pass
w

worried-city-86458

07/25/2021, 11:28 PM
@purple-plumber-90981 Yes, I'm installing jetstack's cert manager and leveraging that where possible in other charts.
p

purple-plumber-90981

07/25/2021, 11:29 PM
yeah i figured it out eventually . . thx… im just trying to give details in my threads for those who follow after and have the same prooblems
w

worried-city-86458

07/25/2021, 11:29 PM
Beware that there is a race condition with cert manager being ready and subsequently issuing certs
p

purple-plumber-90981

07/25/2021, 11:30 PM
a lot of threads in here just have “yeah its fixed now” without any details, which doesnt really help others
w

worried-city-86458

07/25/2021, 11:30 PM
I handle that by issuing a test cert and depending on that. Happy share code (in dotnet/c#).
// cert manager; <https://github.com/jetstack/cert-manager/tree/master/deploy/charts/cert-manager>
Logger.LogDebug("Installing cert manager");
var certManagerCrds = new ConfigGroup("cert-manager-crds",
    new ConfigGroupArgs { Yaml = ReadResource("CertManagerCrds.yaml") },
    new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

var certManagerNs = new Namespace("cert-manager",
    new NamespaceArgs { Metadata = new ObjectMetaArgs { Name = "cert-manager" } },
    new CustomResourceOptions { Provider = k8sProvider });

var certManagerValues =
    new Dictionary<string, object>
    {
        ["prometheus"] = new
        {
            enabled = true,
            servicemonitor = new { enabled = true }
        }
    };

var certManagerChart = new Chart("cert-manager",
    new ChartArgs
    {
        Namespace = "cert-manager",
        FetchOptions = new ChartFetchArgs { Repo = "<https://charts.jetstack.io>" },
        Chart = "cert-manager",
        Version = K8sConfig.CertManagerChartVersion,
        Values = certManagerValues,
        SkipCRDRendering = true
    },
    new ComponentResourceOptions { DependsOn = { certManagerCrds, certManagerNs, kubePrometheusStackCrds }, Provider = k8sProvider });

var certManager = certManagerChart.GetResource<Deployment>("cert-manager", "cert-manager").AsResource();
var certManagerCaInjector = certManagerChart.GetResource<Deployment>("cert-manager-cainjector", "cert-manager").AsResource();
var certManagerWebHook = certManagerChart.GetResource<Deployment>("cert-manager-webhook", "cert-manager").AsResource();

var certManagerTest = new ConfigGroup("cert-manager-test",
    new ConfigGroupArgs { Yaml = RenderTemplate("CertManagerTest.yaml", ReadResource, new { @namespace = "cert-manager" }) },
    new ComponentResourceOptions { DependsOn = { certManager, certManagerCaInjector, certManagerWebHook }, Provider = k8sProvider });

var certManagerTestCert = certManagerTest.GetCustomResource("<http://cert-manager.io/v1/Certificate|cert-manager.io/v1/Certificate>", "cert-manager-test-cert", "cert-manager").AsResource();
//var certManagerTestIssuer = certManagerTest.GetCustomResource("<http://cert-manager.io/v1/Issuer|cert-manager.io/v1/Issuer>", "cert-manager-test-issuer", "cert-manager").AsResource();
p

purple-plumber-90981

07/25/2021, 11:31 PM
so my final pulumi code deploy including the status fix looks like :- 
# for <https://github.com/pulumi/pulumi-kubernetes/issues/800>
# where an upstream helm chart invalidly defines a key for the output only resource "status"
def remove_status(obj):
    if obj["kind"] == "CustomResourceDefinition" and "status" in obj:
        try:
            del obj["status"]
        except KeyError:
            pass


# setup jetstack cert-manager via helm
k8s_h_cert_manager = k8s.helm.v3.Chart(
    "itplat-helm-cert-manager",
    k8s.helm.v3.ChartOpts(
        chart="cert-manager",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://charts.jetstack.io>",
        ),
        namespace='kube-system',
        transformations=[remove_status],
        values={
            "clusterName": pulumi_stack_info['name'],
            "installCRDs": "true",
        },
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)
w

worried-city-86458

07/25/2021, 11:31 PM
// aws load balancer controller; <https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller>
            Logger.LogDebug("Installing aws load balancer controller");
            var awsLbcRole = new RoleX($"{k8sPrefix}-aws-load-balancer-controller",
                new RoleXArgs
                {
                    AssumeRolePolicy = IamHelpers.AssumeRoleForServiceAccount(oidcArn, oidcUrl, "kube-system", "aws-load-balancer-controller", awsProvider),
                    InlinePolicies = { ["policy"] = ReadResource("AwsLoadBalancerPolicy.json") }
                },
                new ComponentResourceOptions { Provider = awsProvider });

            var awsLbcCrds = new ConfigGroup("aws-load-balancer-controller-crds",
                new ConfigGroupArgs { Yaml = ReadResource("AwsLoadBalancerCrds.yaml") },
                new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

            var awsLbcValues = Output.Tuple(clusterName, awsLbcRole.Arn).Apply(((string ClusterName, string RoleArn) tuple) =>
                new Dictionary<string, object>
                {
                    ["clusterName"] = tuple.ClusterName,
                    ["enableCertManager"] = true,
                    ["serviceAccount"] = new { annotations = new Dictionary<string, string> { ["<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"] = tuple.RoleArn } }
                });

            var awsLbcChart = new Chart("aws-load-balancer-controller", // ingress records with <http://alb.ingress.kubernetes.io|alb.ingress.kubernetes.io> annotations depend on chart finalizers
                new ChartArgs
                {
                    Namespace = "kube-system",
                    FetchOptions = new ChartFetchArgs { Repo = "<https://aws.github.io/eks-charts>" },
                    Chart = "aws-load-balancer-controller",
                    Version = K8sConfig.AwsLbcChartVersion,
                    Values = awsLbcValues,
                    SkipCRDRendering = true,
                    ApiVersions = { "<http://admissionregistration.k8s.io/v1|admissionregistration.k8s.io/v1>" }
                },
                new ComponentResourceOptions { DependsOn = { awsLbcCrds, certManagerTestCert }, Provider = k8sProvider });

            var awsLbc = awsLbcChart.GetResource<Deployment>("aws-load-balancer-controller", "kube-system").AsResource();
Note how aws lbc depends on the cert manager test cert
I also install all crds separately as that's the recommend best practice
p

purple-plumber-90981

07/25/2021, 11:33 PM
yeah i did that in python like 
# setup CRD's for aws-load-balancer-controller CRD's
k8s_k_alb_crd = k8s.kustomize.Directory(
    "itplat-alb-crd",
    directory="<https://github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master>",
    transformations=[remove_status],
    opts=pulumi.ResourceOptions(depends_on=[k8s_provider], provider=k8s_provider),
)

# setup aws-load-balancer-controller (ALB) via helm
k8s_h_alb_controller = k8s.helm.v3.Chart(
    "itplat-helm-alb-controller",
    k8s.helm.v3.ChartOpts(
        chart="aws-load-balancer-controller",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://aws.github.io/eks-charts>",
        ),
        namespace='kube-system',
        transformations=[remove_status],
        values={
            "clusterName": pulumi_stack_info['name'],
            "enableCertManager": "true",
        },
    ),
    opts=pulumi.ResourceOptions(depends_on=[k8s_h_cert_manager, k8s_k_alb_crd], provider=k8s_provider),
)
so explicit depends on
ah i see you depend on a created cert… i like it
w

worried-city-86458

07/25/2021, 11:35 PM
Depending on the helm chart is not the same as depending on all chart child resources, hence you currently need to get specific child resources and depend on those.
e.g. cert manager test does: 
var certManagerTest = new ConfigGroup("cert-manager-test",
    new ConfigGroupArgs { Yaml = RenderTemplate("CertManagerTest.yaml", ReadResource, new { @namespace = "cert-manager" }) },
    new ComponentResourceOptions { DependsOn = { certManager, certManagerCaInjector, certManagerWebHook }, Provider = k8sProvider });
p

purple-plumber-90981

07/25/2021, 11:36 PM
im going to use that, awesome, thanks
w

worried-city-86458

07/25/2021, 11:37 PM
cert manager test: 
---
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Issuer
metadata:
  name: cert-manager-test-issuer
  namespace: {{ namespace }}
spec:
  selfSigned: {}
---
apiVersion: <http://cert-manager.io/v1|cert-manager.io/v1>
kind: Certificate
metadata:
  name: cert-manager-test-cert
  namespace: {{ namespace }}
spec:
  dnsNames:
    - <http://example.com|example.com>
  secretName: cert-manager-test
  issuerRef:
    name: cert-manager-test-issuer
p

purple-plumber-90981

07/25/2021, 11:37 PM
sweet, much appreciated
w

worried-city-86458

07/25/2021, 11:37 PM
I render that template replacing the namespace, as shown in the code above.
p

purple-plumber-90981

07/25/2021, 11:39 PM
and in fact i can see i should do similar things in other places in my code where i helm-deploy stuff…
w

worried-city-86458

07/25/2021, 11:40 PM
I'm also using irsa, which I recommend for least privilege best practice. Most aws resources will have irsa setup details, like aws load balancer. Basically, you create an iam role with the least required aws permissions, and associate it with a k8s service account with an annotation.
p

purple-plumber-90981

07/26/2021, 4:01 AM
nice, i will check it out
he @worried-city-86458 i am having issues with cert-manager not creating the deployment… i ’m guessing it is because the service .spec.selector is dependent on “cert-manager.name” and “.Release.Name” . . . did you have to tweek something to have either of these passed in your helmchart for cert-manager ?
ie how you would specify the release to helm cli like 
helm install my-release --namespace cert-manager jetstack/cert-manager
i cant seem to get past 
kubernetes:core/v1:Service (cert-manager/itplat-helm-cert-manager):
    error: 2 errors occurred:
    	* resource cert-manager/itplat-helm-cert-manager was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: 'itplat-helm-cert-manager' timed out waiting to be Ready
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
w

worried-city-86458

07/28/2021, 1:45 AM
I only use what I posted above https://pulumi-community.slack.com/archives/CRFURDVQB/p1627255842432900?thread_ts=1627166292.426000&amp;cid=CRFURDVQB
p

purple-plumber-90981

07/28/2021, 1:48 AM
only difference i see there is SkipCRDRendering = true and your dependence on a pre-existing prometheus stack
can you tell me in your cert-manager service, is you spec selector 
<http://app.kubernetes.io/name|app.kubernetes.io/name>
the same as the logical name for your chart ? (ie “cert-manager”)
im just trying to ensure what pulumi uses as the release name
oh wait no its spec.selector 
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>
that should be release name
w

worried-city-86458

07/28/2021, 1:58 AM
p

purple-plumber-90981

07/28/2021, 2:17 AM
much appreciated
r

rapid-soccer-18092

09/01/2021, 8:05 AM
@worried-city-86458 Hey... so I can't figure this out with the AzureNative SDK: 
var certManager = certManagerChart.GetResource<Deployment>("cert-manager", "cert-manager").AsResource();
var certManagerCaInjector = certManagerChart.GetResource<Deployment>("cert-manager-cainjector", "cert-manager").AsResource();
var certManagerWebHook = certManagerChart.GetResource<Deployment>("cert-manager-webhook", "cert-manager").AsResource();
What type is 
Deployment
and is 
AsResource()
an extension method from somewhere?
w

worried-city-86458

09/01/2021, 8:09 AM
Pulumi.Kubernetes.Apps.V1.Deployment
and yes AsResource is an extension method to hide the ugly cast via apply
✅ 1
(Needed because subsequent 
DependsOn
is type 
InputList<Resource>
)
r

rapid-soccer-18092

09/01/2021, 8:19 AM
Ok great - thank you
w

witty-belgium-75866

11/08/2021, 9:33 AM
hi, what was the solution to the original question? I can not access the posted solution 😞
( any time I'm executing 
pulumi up
, some of the resources of the alb ingress controller are being recreated 😞 )
4 Views
#kubernetesJoin Slack
Powered by