https://pulumi.com logo
Powered by
Title
w

worried-city-86458

06/16/2021, 5:34 AM
I'm installing the aws load balancer controller via the helm chart. Every time I run update it wants to create-replacement the tls secret.
dotnet/c# code: 
// aws load balancer; <https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller>
Logger.LogDebug("Installing aws load balancer");
var awsLbcRole = new RoleX($"{k8sPrefix}-aws-load-balancer-controller",
    new RoleXArgs
    {
        AssumeRolePolicy = IamHelpers.AssumeRoleForServiceAccount(oidcArn, oidcUrl, "kube-system", "aws-load-balancer-controller", awsProvider),
        InlinePolicies = { ["policy"] = ReadResource("AwsLoadBalancerPolicy.json") }
    },
    new ComponentResourceOptions { Provider = awsProvider });

var awsLbcCrds = new ConfigGroup("aws-load-balancer-controller-crds",
    new ConfigGroupArgs { Yaml = ReadResource("AwsLoadBalancerCrds.yaml") },
    new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

var awsLbcValues = Output.Tuple(clusterName, awsLbcRole.Arn).Apply(((string ClusterName, string RoleArn) tuple) =>
    new Dictionary<string, object>
    {
        ["clusterName"] = tuple.ClusterName,
        ["serviceAccount"] = new { annotations = new Dictionary<string, string> { ["<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"] = tuple.RoleArn } }
    });

new Chart("aws-load-balancer-controller",
    new ChartArgs
    {
        Namespace = "kube-system",
        FetchOptions = new ChartFetchArgs { Repo = "<https://aws.github.io/eks-charts>" },
        Chart = "aws-load-balancer-controller",
        Version = K8sConfig.AwsLbcChartVersion,
        Values = awsLbcValues,
        SkipCRDRendering = true
    },
    new ComponentResourceOptions { DependsOn = awsLbcCrds, Provider = k8sProvider });
Changes:
 
    Type                                                                   Name                              Operation
>   pulumi:pulumi:StackReference                                           pharos/aws-eks/alpha              read
++  kubernetes:core:Secret                                                 kube-system/aws-load-balancer-tls create-replacement
~   kubernetes:<http://admissionregistration.k8s.io:ValidatingWebhookConfiguration|admissionregistration.k8s.io:ValidatingWebhookConfiguration> aws-load-balancer-webhook         update
~   kubernetes:<http://admissionregistration.k8s.io:MutatingWebhookConfiguration|admissionregistration.k8s.io:MutatingWebhookConfiguration>   aws-load-balancer-webhook         update
 
Resources:
    +-replace 1
    ~ update 2
    72 unchanged
 
Duration: 6s
b

bored-table-20691

06/16/2021, 5:36 AM
@worried-city-86458 what’s the “diff” for the replaced secret?
See an example here: https://pulumi-community.slack.com/archives/CRFURDVQB/p1619251600226800
w

worried-city-86458

06/16/2021, 5:44 AM
This is the diff in the web ui: 
kube-system/aws-load-balancer-tls (kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret|helm.sh/v3:Chart$kubernetes:core/v1:Secret>)
++ kubernetes:core/v1:Secret (create-replacement)
    [id=kube-system/aws-load-balancer-tls]
    [urn=urn:pulumi:alpha::k8s::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls|helm.sh/v3:Chart$kubernetes:core/v1:Secret::kube-system/aws-load-balancer-tls>]
    __initialApiVersion: "v1"
    __inputs           : {
        apiVersion: "v1"
        data      : "[secret]"
        kind      : "Secret"
        metadata  : {
            labels   : {
                <http://app.kubernetes.io/instance|app.kubernetes.io/instance>  : "aws-load-balancer-controller"
                <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: "pulumi"
                <http://app.kubernetes.io/name|app.kubernetes.io/name>      : "aws-load-balancer-controller"
                <http://app.kubernetes.io/version|app.kubernetes.io/version>   : "v2.2.0"
                <http://helm.sh/chart|helm.sh/chart>               : "aws-load-balancer-controller-1.2.2"
            }
            name     : "aws-load-balancer-tls"
            namespace: "kube-system"
        }
        type      : "<http://kubernetes.io/tls|kubernetes.io/tls>"
    }
    metadata           : {
        annotations      : {
            <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: "[secret]"
        }
        creationTimestamp: "2021-06-16T04:58:27Z"
        managedFields    : [
            [0]: {
                apiVersion: "v1"
                fieldsType: "FieldsV1"
                fieldsV1  : {
                    f:data    : {
                        .        : {}
                        f:ca.crt : {}
                        f:tls.crt: {}
                        f:tls.key: {}
                    }
                    f:metadata: {
                        f:annotations: {
                            .                                                 : {}
                            f:<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: {}
                        }
                        f:labels     : {
                            .                             : {}
                            f:<http://app.kubernetes.io/instance|app.kubernetes.io/instance>  : {}
                            f:<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: {}
                            f:<http://app.kubernetes.io/name|app.kubernetes.io/name>      : {}
                            f:<http://app.kubernetes.io/version|app.kubernetes.io/version>   : {}
                            f:<http://helm.sh/chart|helm.sh/chart>               : {}
                        }
                    }
                    f:type    : {}
                }
                manager   : "pulumi-resource-kubernetes.exe"
                operation : "Update"
                time      : "2021-06-16T04:58:27Z"
            }
        ]
        resourceVersion  : "55587"
        uid              : "f80096e8-f6dc-434e-8339-20947915367d"
    }
This is the diff in the cli (using dotnet automation api):
Pulumi.log
b

bored-table-20691

06/16/2021, 5:55 AM
My guess is the helm chart is somehow generating a new TLS cert every time
w

worried-city-86458

06/16/2021, 6:01 AM
Right. https://github.com/aws/eks-charts/blob/master/stable/aws-load-balancer-controller/templates/webhook.yaml#L133
https://github.com/aws/eks-charts/blob/master/stable/aws-load-balancer-controller/templates/webhook.yaml#L1
b

bored-table-20691

06/16/2021, 6:04 AM
Yeah. So probably the easiest would be to apply a trasnform in Pulumi to the Helm chart and to not deploy this secret, or to override it with your known values.
w

worried-city-86458

06/16/2021, 6:05 AM
Maybe enabling cert manager would help too, assuming cert manager would only generate the cert once.
b

bored-table-20691

06/16/2021, 6:06 AM
Generally yes
b

billowy-army-68599

06/16/2021, 6:26 AM
@worried-city-86458 you'll either need to: - use 
ignoreChanges
on the fields - generate your own secrets with pulumi-tls
it generates new secrets each time because of a helm helper function that relies on server side logic
w

worried-city-86458

06/16/2021, 6:27 AM
Thanks. Yeah I saw that when I went digging.
b

billowy-army-68599

06/16/2021, 6:28 AM
generating your own tls secrets can be found here: (search for all references to tls) https://github.com/jaxxstorm/pulumi-awsloadbalancercontroller/blob/main/provider/pkg/provider/awsloadbalancercontroller.go#L364 ignoreChanges is easier
w

worried-city-86458

06/16/2021, 6:37 AM
The chart supported cert manager, so I enabled that and all good now.
👍 1
https://github.com/aws/eks-charts/blob/d442bce67889527510fa170a8b3dfa6be4015248/stable/aws-load-balancer-controller/templates/webhook.yaml#L145-L169
Thanks for your help, guys!
4 Views
#kubernetesJoin Slack
Powered by