https://pulumi.com logo
Join Slack
Powered by
After struggling with two different helm charts (c...
# general
i
After struggling with two different helm charts (cert-manager and kubernetes-dashboard), I’m finding that it is far easier to take the approach of using the 
ConfigFile
with the published manifests, and optionally using a 
transformation
. I had assumed the helm approach to be easier - but surprised to find it the other way around. If anyone has issues with those two components, let me know. I put in quite a bit of time learning the ins-and-outs of pulumi with them and getting them running properly, which actually ended up being quite easy in hindsight. My new approach to anything will be to look at the component and manifests first, then go from there.
c
@important-leather-28796 fascinating, how are you getting the YAML?
just 
helm template
?
i
via raw url from the recommended install docs for each (avoiding helm altogether) e.g.
Copy code
// Install the CustomResourceDefinitions and cert-manager itself
    const certManager = new k8s.yaml.ConfigFile(
      '<https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/cert-manager.yaml>',
      undefined,
      { parent: this },
    )
It feels like I never should have tried helm when a component’s release publishes the raw manifest. Pulumi allows the simple customization of anything I need so no need to obfuscate through another helm developer’s interpretation of the original manifest (or that’s the way I feel)
doubling the toolchain just caused me to chase unnecessary things - a secret on the k8 dashboard and namespace/webhook issue on the cert-manager
raw manifests pretty much worked out of the box.
So - not much of a revelation - just that helm won’t be my first look anymore
c
helm also has a bunch of small gotchas, like ~most charts re-generate secrets every invocation of the chart.
i
it just seems like unnecessary complexity in hindsight. I wish I would have known that but not sure how you could introduce that thought to other users diplomatically.
o
also having some helm issues
c
cc @breezy-hamburger-69619 🙂
m
I have actually had really good luck using helm charts with pulumi.
o
for example deploying a newrelic chart. in the function, create a namespace, pass namespace as dependsOn to subsquent helm install of the kube-state-metrics and newrelic charts. Even if changing the namespace on which the helm object depends, it does not redeploy the app
(this is in a case where we initially deployed the apps to the ‘wrong’ namespace)
c
@orange-tailor-85423 
dependsOn
does not set the namespace
o
correct
c
it just causes the namespace to be provisioned first.
o
I hard-coded that change as well in the chart
m
With one small exception also related to namespaces. It seems setting the 
namespace
input field does not actually cause the resources to end up there. Instead I do this:
Copy code
transformations: [(y) => {
        y.metadata.namespace = 'kube-system'; // eslint-disable-line
      }],
c
yeah, we will have a fix for that very soon.
it should definitely re-provision the chart if you’re changing the namespace of the resources in it.
m
I have found that to be true, just requiring the 
transformations
hack to actually change them.
o
wow
maybe this our issue then
m
Also should still set 
namespace: kube-system
on the 
k8s.helm.v2.Chart
, since the chart may interpolate on 
.Release.namespace
For now, need both, sounds like fix incoming to only need 
namespace: kube-system
, and no 
transformations
type hack
o
Tim, larger example of the transformations hack?
m
That's literally all I add if I want to move everything to a namesapce
do you mean a full resource?
Copy code
const chartIgnored = new k8s.helm.v2.Chart('ingress', {
      repo: 'incubator',
      chart: 'aws-alb-ingress-controller',
      version: '0.1.4',
      namespace: 'kube-system',
      values: {
        clusterName: name,
        autoDiscoverAwsRegion: true,
        autoDiscoverAwsVpcID: true,
      },
      transformations: [(y) => {
        y.metadata.namespace = 'kube-system'; // eslint-disable-line
      }],
    }, defaultOpts);
c
so we will eventually fix the 
namespace: kube-system
thing, but this week is very likely to fix the transforms stuff.
the 
namespace: kube-system
thing is so, so annoying, because it’s something helm shoudl be doing itself, and we’ll have to work around.
m
Huh, I don't understand what the transforms fix is.
What is the issue there?
One other interesting helm chart hack I have had to do that may be of interest to people:
Copy code
transformations: [(y) => {
        // Fix stupid name changing
        if (y.kind === 'Pod' && y.metadata.name.indexOf('jenkins-ui-test') !== -1) {
          y.metadata.name = 'jenkins-ui-test'; // eslint-disable-line
        }
      }],
The jenkins helm chart has a pod that gets random characters appended to its name, causing a constant diff. This fixes it.
o
code:
Copy code
export function Install(
    name: string,
    kubestateMetricsVersion: string,
    version: string,
    licenseKey: string,
    env: string,
    cluster: k8s.Provider
): k8s.helm.v2.Chart {
    const namespace = new k8s.core.v1.Namespace(
        `newrelic3`,
        {
            metadata: { name: `newrelic3` }
        },
        { provider: cluster }
    );

    new k8s.helm.v2.Chart(
        name,
        {
            repo: 'stable',
            chart: 'kube-state-metrics',
            version: kubestateMetricsVersion,
            namespace: 'newrelic3'
        },
        {
            dependsOn: [namespace],
            providers: { kubernetes: cluster }
        }
    );
    return new k8s.helm.v2.Chart(
        `${name}-infra`,
        {
            repo: 'stable',
            chart: 'newrelic-infrastructure',
            namespace: 'newrelic3',
            version,
            values: { licenseKey, cluster: env }
        },
        {
            dependsOn: [namespace],
            providers: { kubernetes: cluster }
        }
    );
}
this is going from ‘newrelic’ to ‘newrelic2’ to ‘newrelic3’
Copy code
Type                                                           Name                                    Plan       Info
     pulumi:pulumi:Stack                                            drogon-casey-robertson
 >   ├─ pulumi:pulumi:StackReference                                networkStorageCluster                   read
 +   ├─ kubernetes:core:Namespace                                   newrelic3                               create
     ├─ kubernetes:helm.sh:Chart                                    newrelic-infra
 ~   │  └─ kubernetes:<http://rbac.authorization.k8s.io:ClusterRoleBinding|rbac.authorization.k8s.io:ClusterRoleBinding>  newrelic-infra-newrelic-infrastructure  update     [diff: ~subjects]
     ├─ kubernetes:helm.sh:Chart                                    newrelic
 ~   │  └─ kubernetes:<http://rbac.authorization.k8s.io:ClusterRoleBinding|rbac.authorization.k8s.io:ClusterRoleBinding>  newrelic-kube-state-metrics             update     [diff: ~subjects]
 -   └─ kubernetes:core:Namespace                                   newrelic2                               delete

Resources:
    + 1 to create
    ~ 2 to update
    - 1 to delete
creates new namespace but nothing happens to the deployed app
m
Add the 
transformations
hack:
Copy code
return new k8s.helm.v2.Chart(
        `${name}-infra`,
        {
            repo: 'stable',
            chart: 'newrelic-infrastructure',
            namespace: 'newrelic3',
            version,
            values: { licenseKey, cluster: env },
            transformations: [(y) => {
                y.metadata.namespace = 'newrelic3'; // eslint-disable-line
            }],
        },
        {
            dependsOn: [namespace],
            providers: { kubernetes: cluster }
        }
    );
@orange-tailor-85423 ^
c
@millions-judge-24978 I mean that transforms don't allow you to put output<t> in them right now. will fix that
but also you should just be able to set the namespace in chart opts and have it work
m
Ah I see, when does that become an issue? I haven't noticed any problem.
I guess just some case in which you need something more dynamic than what I'm doing in there
c
the release namespace issue you mentioned
I'm on phone
I think you mentioned it. it's the option that helm template populates
o
working now Tim - thanks so much
💯 1
I need to read up on that - I’m not a helm expert
m
Oh my only issue is the namespace option in chart inputs does not move resources
o
is there an open issue to track this?
Open in Slack
PreviousNext
Powered by