https://pulumi.com logo
Powered by
i

important-leather-28796

03/12/2019, 10:51 PM
After struggling with two different helm charts (cert-manager and kubernetes-dashboard), I’m finding that it is far easier to take the approach of using the 
ConfigFile
with the published manifests, and optionally using a 
transformation
. I had assumed the helm approach to be easier - but surprised to find it the other way around. If anyone has issues with those two components, let me know. I put in quite a bit of time learning the ins-and-outs of pulumi with them and getting them running properly, which actually ended up being quite easy in hindsight. My new approach to anything will be to look at the component and manifests first, then go from there.
c

creamy-potato-29402

03/12/2019, 10:52 PM
@important-leather-28796 fascinating, how are you getting the YAML?
just 
helm template
?
i

important-leather-28796

03/12/2019, 10:54 PM
via raw url from the recommended install docs for each (avoiding helm altogether) e.g.
Copy code
// Install the CustomResourceDefinitions and cert-manager itself
    const certManager = new k8s.yaml.ConfigFile(
      '<https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/cert-manager.yaml>',
      undefined,
      { parent: this },
    )
It feels like I never should have tried helm when a component’s release publishes the raw manifest. Pulumi allows the simple customization of anything I need so no need to obfuscate through another helm developer’s interpretation of the original manifest (or that’s the way I feel)
doubling the toolchain just caused me to chase unnecessary things - a secret on the k8 dashboard and namespace/webhook issue on the cert-manager
raw manifests pretty much worked out of the box.
So - not much of a revelation - just that helm won’t be my first look anymore
c

creamy-potato-29402

03/12/2019, 11:00 PM
helm also has a bunch of small gotchas, like ~most charts re-generate secrets every invocation of the chart.
i

important-leather-28796

03/12/2019, 11:02 PM
it just seems like unnecessary complexity in hindsight. I wish I would have known that but not sure how you could introduce that thought to other users diplomatically.
o

orange-tailor-85423

03/12/2019, 11:12 PM
also having some helm issues
c

creamy-potato-29402

03/12/2019, 11:13 PM
cc @breezy-hamburger-69619 🙂
m

millions-judge-24978

03/12/2019, 11:13 PM
I have actually had really good luck using helm charts with pulumi.
o

orange-tailor-85423

03/12/2019, 11:13 PM
for example deploying a newrelic chart. in the function, create a namespace, pass namespace as dependsOn to subsquent helm install of the kube-state-metrics and newrelic charts. Even if changing the namespace on which the helm object depends, it does not redeploy the app
(this is in a case where we initially deployed the apps to the ‘wrong’ namespace)
c

creamy-potato-29402

03/12/2019, 11:14 PM
@orange-tailor-85423 
dependsOn
does not set the namespace
o

orange-tailor-85423

03/12/2019, 11:14 PM
correct
c

creamy-potato-29402

03/12/2019, 11:14 PM
it just causes the namespace to be provisioned first.
o

orange-tailor-85423

03/12/2019, 11:14 PM
I hard-coded that change as well in the chart
m

millions-judge-24978

03/12/2019, 11:14 PM
With one small exception also related to namespaces. It seems setting the 
namespace
input field does not actually cause the resources to end up there. Instead I do this:
Copy code
transformations: [(y) => {
        y.metadata.namespace = 'kube-system'; // eslint-disable-line
      }],
c

creamy-potato-29402

03/12/2019, 11:14 PM
yeah, we will have a fix for that very soon.
it should definitely re-provision the chart if you’re changing the namespace of the resources in it.
m

millions-judge-24978

03/12/2019, 11:15 PM
I have found that to be true, just requiring the 
transformations
hack to actually change them.
o

orange-tailor-85423

03/12/2019, 11:15 PM
wow
maybe this our issue then
m

millions-judge-24978

03/12/2019, 11:16 PM
Also should still set 
namespace: kube-system
on the 
k8s.helm.v2.Chart
, since the chart may interpolate on 
.Release.namespace
For now, need both, sounds like fix incoming to only need 
namespace: kube-system
, and no 
transformations
type hack
o

orange-tailor-85423

03/12/2019, 11:16 PM
Tim, larger example of the transformations hack?
m

millions-judge-24978

03/12/2019, 11:17 PM
That's literally all I add if I want to move everything to a namesapce
do you mean a full resource?
Copy code
const chartIgnored = new k8s.helm.v2.Chart('ingress', {
      repo: 'incubator',
      chart: 'aws-alb-ingress-controller',
      version: '0.1.4',
      namespace: 'kube-system',
      values: {
        clusterName: name,
        autoDiscoverAwsRegion: true,
        autoDiscoverAwsVpcID: true,
      },
      transformations: [(y) => {
        y.metadata.namespace = 'kube-system'; // eslint-disable-line
      }],
    }, defaultOpts);
c

creamy-potato-29402

03/12/2019, 11:17 PM
so we will eventually fix the 
namespace: kube-system
thing, but this week is very likely to fix the transforms stuff.
the 
namespace: kube-system
thing is so, so annoying, because it’s something helm shoudl be doing itself, and we’ll have to work around.
m

millions-judge-24978

03/12/2019, 11:18 PM
Huh, I don't understand what the transforms fix is.
What is the issue there?
One other interesting helm chart hack I have had to do that may be of interest to people:
Copy code
transformations: [(y) => {
        // Fix stupid name changing
        if (y.kind === 'Pod' && y.metadata.name.indexOf('jenkins-ui-test') !== -1) {
          y.metadata.name = 'jenkins-ui-test'; // eslint-disable-line
        }
      }],
The jenkins helm chart has a pod that gets random characters appended to its name, causing a constant diff. This fixes it.
o

orange-tailor-85423

03/12/2019, 11:21 PM
code:
Copy code
export function Install(
    name: string,
    kubestateMetricsVersion: string,
    version: string,
    licenseKey: string,
    env: string,
    cluster: k8s.Provider
): k8s.helm.v2.Chart {
    const namespace = new k8s.core.v1.Namespace(
        `newrelic3`,
        {
            metadata: { name: `newrelic3` }
        },
        { provider: cluster }
    );

    new k8s.helm.v2.Chart(
        name,
        {
            repo: 'stable',
            chart: 'kube-state-metrics',
            version: kubestateMetricsVersion,
            namespace: 'newrelic3'
        },
        {
            dependsOn: [namespace],
            providers: { kubernetes: cluster }
        }
    );
    return new k8s.helm.v2.Chart(
        `${name}-infra`,
        {
            repo: 'stable',
            chart: 'newrelic-infrastructure',
            namespace: 'newrelic3',
            version,
            values: { licenseKey, cluster: env }
        },
        {
            dependsOn: [namespace],
            providers: { kubernetes: cluster }
        }
    );
}
this is going from ‘newrelic’ to ‘newrelic2’ to ‘newrelic3’
Copy code
Type                                                           Name                                    Plan       Info
     pulumi:pulumi:Stack                                            drogon-casey-robertson
 >   ├─ pulumi:pulumi:StackReference                                networkStorageCluster                   read
 +   ├─ kubernetes:core:Namespace                                   newrelic3                               create
     ├─ kubernetes:helm.sh:Chart                                    newrelic-infra
 ~   │  └─ kubernetes:<http://rbac.authorization.k8s.io:ClusterRoleBinding|rbac.authorization.k8s.io:ClusterRoleBinding>  newrelic-infra-newrelic-infrastructure  update     [diff: ~subjects]
     ├─ kubernetes:helm.sh:Chart                                    newrelic
 ~   │  └─ kubernetes:<http://rbac.authorization.k8s.io:ClusterRoleBinding|rbac.authorization.k8s.io:ClusterRoleBinding>  newrelic-kube-state-metrics             update     [diff: ~subjects]
 -   └─ kubernetes:core:Namespace                                   newrelic2                               delete

Resources:
    + 1 to create
    ~ 2 to update
    - 1 to delete
creates new namespace but nothing happens to the deployed app
m

millions-judge-24978

03/12/2019, 11:23 PM
Add the 
transformations
hack:
Copy code
return new k8s.helm.v2.Chart(
        `${name}-infra`,
        {
            repo: 'stable',
            chart: 'newrelic-infrastructure',
            namespace: 'newrelic3',
            version,
            values: { licenseKey, cluster: env },
            transformations: [(y) => {
                y.metadata.namespace = 'newrelic3'; // eslint-disable-line
            }],
        },
        {
            dependsOn: [namespace],
            providers: { kubernetes: cluster }
        }
    );
@orange-tailor-85423 ^
c

creamy-potato-29402

03/12/2019, 11:25 PM
@millions-judge-24978 I mean that transforms don't allow you to put output<t> in them right now. will fix that
but also you should just be able to set the namespace in chart opts and have it work
m

millions-judge-24978

03/12/2019, 11:27 PM
Ah I see, when does that become an issue? I haven't noticed any problem.
I guess just some case in which you need something more dynamic than what I'm doing in there
c

creamy-potato-29402

03/12/2019, 11:28 PM
the release namespace issue you mentioned
I'm on phone
I think you mentioned it. it's the option that helm template populates
o

orange-tailor-85423

03/12/2019, 11:29 PM
working now Tim - thanks so much
💯 1
I need to read up on that - I’m not a helm expert
m

millions-judge-24978

03/12/2019, 11:30 PM
Oh my only issue is the namespace option in chart inputs does not move resources
o

orange-tailor-85423

03/12/2019, 11:59 PM
is there an open issue to track this?
Powered by