https://pulumi.com logo
#general
Title
# general
e

early-musician-41645

10/29/2018, 10:26 PM
Following the install of a kubernetes cluster I'd like to automate deployment of a few helm charts. The charts are hosted in a private helm repo hosted in S3 that I normally access with
helm repo add my-repo <s3://my-helm-repo/charts>
. Given the example service deployment (copied below) how do I specify custom repo endpoints using the S3 plugin?
Copy code
// Sample copied from <https://github.com/pulumi/examples/blob/master/kubernetes-ts-helm-wordpress/index.ts>

// Deploy the latest version of the stable/wordpress chart.
const wordpress = new k8s.helm.v2.Chart("wpdev", {
    repo: "stable",
    version: "2.1.3",
    chart: "wordpress"
});
c

creamy-potato-29402

10/29/2018, 10:29 PM
@early-musician-41645 this is a thing I did not know you could do 🙂
e

early-musician-41645

10/29/2018, 10:30 PM
There's a Helm S3 plugin that allows accessing an s3-based chart repo
c

creamy-potato-29402

10/29/2018, 10:30 PM
@early-musician-41645 I’ll file an issue and fix this release.
m

microscopic-florist-22719

10/29/2018, 10:30 PM
There's a chance that this just works if you configure the repo before running Pulumi
c

creamy-potato-29402

10/29/2018, 10:31 PM
oh oh
it’s a plugin
hmm
e

early-musician-41645

10/29/2018, 10:31 PM
ah, good point... so Pulumi is using the local Helm, right?
m

microscopic-florist-22719

10/29/2018, 10:31 PM
correct
e

early-musician-41645

10/29/2018, 10:31 PM
If that's the case then I can just do the
helm repo add
before calling
pulumi up
. I'll try it out
c

creamy-potato-29402

10/29/2018, 10:32 PM
yeah.
hmm, not sure how we should handle plugins….
e

early-musician-41645

10/29/2018, 10:32 PM
Question is - how do I reference the repo in the code? Normally I do the command I have above, but then to reference the repo/package I do
helm install my-repo/my-chart
So in the example, can I just do
repo: my-repo/my-chart
?
c

creamy-potato-29402

10/29/2018, 10:34 PM
@early-musician-41645
{repo: "my-repo", chart: "my-chart"}
?
the opts have a field for repo
e

early-musician-41645

10/29/2018, 10:34 PM
do you have the link to the opts code?
does it support tiller-namespace and namespace?
and also, how to pass in values overrides, e.g.
--values
and
--set-string
?
e

early-musician-41645

10/29/2018, 10:37 PM
hmmm... specifically I want to pass in a file for values, e.g.
--values $filepath
c

creamy-potato-29402

10/29/2018, 10:41 PM
@early-musician-41645 you mean from shell?
or…
I’m not sure I understand.
e

early-musician-41645

10/29/2018, 10:43 PM
Typically I pass in values by giving a filepath. Here's a more complete example of things I need to do:
Copy code
cmd="helm upgrade -i backgrounder $HELM_REPO/backgrounder \
      --tiller-namespace $namespace \
      --namespace $namespace \
      --values $podsettings_file \
      --set global.podSettings.podSettingsVersion=$configuration_version \
      --set images.tag=$service_version
      --set-string global.source.commit=$CI_COMMIT_SHA \
      --set-string global.source.repo=$CI_COMMIT_REF_SLUG \
      --set-string global.deploymentId=$deployment_id"
So the
--set-string
is handles by the
values
field in the pulumi code, but how to pass in a filepath instead?
Also, same question about setting the
namespace
and
tiller-namespace
And then the followup - how do I install tiller into a specific namespace?
c

creamy-potato-29402

10/29/2018, 10:49 PM
@early-musician-41645 we pick up the default values file, but not a custom file
if I understand correctly what
--tiller-namespace
does… we don’t actually use tiller at all, so I think this flag is meaningless to us?
e

early-musician-41645

10/29/2018, 10:50 PM
You use helm without Tiller?
c

creamy-potato-29402

10/29/2018, 10:50 PM
Yeah.
e

early-musician-41645

10/29/2018, 10:51 PM
Does that work?
c

creamy-potato-29402

10/29/2018, 10:51 PM
we are basically equivalent to
helm template
It works for the vast marjority of charts.
but, anyway, Helm 3 won’t have tiller so we’re consistent with where Helm is going.
e

early-musician-41645

10/29/2018, 10:52 PM
The scenario I have is that we have some services with LOTS of environment-specific configuration, and we have lots of environments, and lots of services. When deploying a service I download a config, specific to an environment, and versioned, and then feed that into the helm command via
--values
c

creamy-potato-29402

10/29/2018, 10:53 PM
that makes sense, and that should be doable with Pulumi too
e

early-musician-41645

10/29/2018, 10:53 PM
We model environments using namespaces. And to track services running in a single namespace we deploy tiller into that one namespace
Hence the need to
--tiller-namespace
c

creamy-potato-29402

10/29/2018, 10:54 PM
mmmmmm
e

early-musician-41645

10/29/2018, 10:54 PM
If there's a better practice then I'm open to it
c

creamy-potato-29402

10/29/2018, 10:56 PM
@early-musician-41645 for using Helm, I’m not sure there is
that’s a super interesting use case
the way Pulumi works, we basically run
helm template
and apply the result.
--tiller-namespace
basically specifies which namespace tiller resides in, right? but it still has access to probably the entire cluster?
unless you configured it differently
e

early-musician-41645

10/29/2018, 10:59 PM
Okay, I may be able to work around this by forming the
values
from a file input and injecting them at runtime
I've configured each tiller to use a service account specific to the namespace
so it's restricted
c

creamy-potato-29402

10/29/2018, 11:00 PM
So the “Pulumi way” is that you should parameterize your app on namespaces. This is somewhat different from the Helm way.
this is hard with Helm. It’s easier with Pulumi.
e

early-musician-41645

10/29/2018, 11:01 PM
Yeah, unfortunately that's not an option. The "environment" is the top-level resource in our deployments, not the app. It's because we are deploying a distributed-monolith where apps must be deployed together...
We have something like 40 sandbox "environments", isolated by namespace, and each environment has something like 20 apps
c

creamy-potato-29402

10/29/2018, 11:04 PM
hmm, not sure I understand. you’re trying to deploy the application to a specific namespace based on where tiller is located?
e

early-musician-41645

10/29/2018, 11:09 PM
I'm trying to deploy the app to a namespace based on need. E.g. we need Foo app, upgraded to v123, in the Bar environment. We've added another layer of isolation by running a tiller instance in each environment to track that environment's deployments
c

creamy-potato-29402

10/29/2018, 11:13 PM
I’ve got to go, but I’ll be back in a bit
@early-musician-41645 so, getting back to this, I’m not sure yet whether we’ll ever support tiller.
Using tiller precludes getting the per-resource CLI/CI experience that I think of when I think of Pulumi Kubernetes integration…
In other words I’m not sure what value Pulumi adds if you are using Tiller.
I’m happy to change my mind about this though if there is something I’m missing…
e

early-musician-41645

10/30/2018, 6:35 PM
In terms of scenarios for locking down access to the kubernetes cluster, what's the model for Pulumi? Here's a for-instance: https://medium.com/@amimahloof/how-to-setup-helm-and-tiller-with-rbac-and-namespaces-34bf27f7d3c3 In that article the namespaces are controlled with service accounts and tiller is installed under that service account to enable RBAC control, e.g. devs are only allowed to deploy via the "dev" service account which grants access to tiller in a set of namespaces, but admins can also deploy to kube-system, etc. Via Pulumi, how can I set up that type of RBAC control for deployments?
c

creamy-potato-29402

10/30/2018, 6:40 PM
@early-musician-41645 Pulumi uses the official kubernetes client under the covers … so the story is basically identical to however you would do this with
kubectl
Probably you’d have a service account allocated for various roles (e.g., CI/CD, admin, whatever), and the access for those SAs would be defined by a set of RBAC definitions.
Managing RBAC with Pulumi is very natural in this sense — you don’t need to do anything out of the ordinary because Pulumi is just a normal Kubernetes client.
e

early-musician-41645

10/30/2018, 6:49 PM
Thanks for explaining it. That makes sense. I'll discuss with my team about it to see what we can work with. Is there a doc or link detailing best practices or examples of typical setups like this with Pulumi?
c

creamy-potato-29402

10/30/2018, 7:02 PM
@early-musician-41645 there’s a bug here: https://github.com/pulumi/examples/issues/158
It’s slated for the next couple of weeks.
e

early-musician-41645

10/30/2018, 10:41 PM
Okay, that works