https://pulumi.com logo
#general
Title
# general
q

quiet-wolf-18467

11/02/2018, 11:44 PM
Is it possible to dynamically configure AWS, GCP, or K8s?
c

creamy-potato-29402

11/02/2018, 11:50 PM
@quiet-wolf-18467 what do you mean when you say “dynamically”?
q

quiet-wolf-18467

11/02/2018, 11:51 PM
I'm trying to setup "Pulumi Packs" that provide reusable components for people to use
I want to allow passing different GCP or other accounts to the function
c

creamy-potato-29402

11/02/2018, 11:55 PM
Hmm, I still am not sure I understand. You mean, passing service accounts around?
maybe it would help if you had an example of the code you’d like to write?
q

quiet-wolf-18467

11/03/2018, 12:01 AM
Say I want to deploy an pod to 2 different Kubernetes clusters. Is that possible?
c

creamy-potato-29402

11/03/2018, 12:02 AM
yeah, sure. In the case of Kubernetes you have two options.
(1) the k8s provider picks up whatever is in your kubeconfig file, so if you can tell the kubernetes provider to use a different context, which points at a different cluster. For this to be sensible you’d need one stack per cluster, though.
q

quiet-wolf-18467

11/03/2018, 12:04 AM
And if I wanted it in a single stack?
c

creamy-potato-29402

11/03/2018, 12:04 AM
(2) in the case of a high-order provider, you’d just specify two different providers. So say you boot up an AKS and GKE cluster in the same app. You’d write a
for
loop over both.
q

quiet-wolf-18467

11/03/2018, 12:04 AM
So providers can be configured in code? That's what I'm after. Can the same be done for AWS, Azure, and GCS?
GCP, sorry
c

creamy-potato-29402

11/03/2018, 12:04 AM
hmm, you’d probably want to have 2 providers, one for each cluster.
yeah.
That I do not know.
c

creamy-potato-29402

11/03/2018, 12:05 AM
@microscopic-florist-22719?
q

quiet-wolf-18467

11/03/2018, 12:05 AM
It's super early. But I want to provide something that is configurable like that
c

creamy-potato-29402

11/03/2018, 12:05 AM
I’d love to help.
Pat is the guy to ask though.
q

quiet-wolf-18467

11/03/2018, 12:05 AM
I plan to provide for TypeScript, Python, and Go
And I want to write functions for Kubernetes deployments to check InfluxDB and Prometheus for linear regression and automated rollbscks
So it's all plug and play
c

creamy-potato-29402

11/03/2018, 12:06 AM
oooo have you seen our staged rollout example for k8s?
if not, check it out.
g

glamorous-printer-66548

11/03/2018, 12:09 AM
@quiet-wolf-18467 funny to see that. I actually built a pulumi package to setup gitlab runner as well. In my case it’s setting up a gke cluster from scratch and installs an auto scaling gitlab-runner onto it. Maybe I should open source it - it’s pretty generic already.
q

quiet-wolf-18467

11/03/2018, 12:10 AM
I've seen the stages roll out. But it's not that reusable.
c

creamy-potato-29402

11/03/2018, 12:11 AM
that’s true.
you mean in the sense that it’s not built like a library?
q

quiet-wolf-18467

11/03/2018, 12:12 AM
Exactly
c

creamy-potato-29402

11/03/2018, 12:13 AM
maybe I should make it a library
q

quiet-wolf-18467

11/03/2018, 12:13 AM
I'd like to just import the function 😄
c

creamy-potato-29402

11/03/2018, 12:13 AM
yeah
it was meant as an example, but maybe it’s worth doing as a library
q

quiet-wolf-18467

11/03/2018, 12:14 AM
@glamorous-printer-66548 I'd love to see it
I think there's many thing that can be shared this way. I think Pulumi can replace Helm. While I know Pulumi can deploy Helm charts, it's also much better to codify than Go Templates
👍 1
I hope I'm thinking things that are already on the teams minds and not going too far down the crazy path 😃
m

microscopic-florist-22719

11/03/2018, 12:24 AM
Late to the party here, but yes you can do this.
With one catch: neither Python nor Go expose first-class providers yet.
c

creamy-potato-29402

11/03/2018, 12:25 AM
@quiet-wolf-18467 well, on the k8s side, my goal is to make this situation really easy, yes.
m

microscopic-florist-22719

11/03/2018, 12:25 AM
In TypeScript, you just do
new aws.Provider(“name”, { inputs })
like with any other resource
g

glamorous-printer-66548

11/03/2018, 12:26 AM
@quiet-wolf-18467 absolutely agree. Would love to have a helm-like ecosystem for pulumi packages 🙂
c

creamy-potato-29402

11/03/2018, 12:26 AM
i.e. to make it really easy to author libraries you can use anywhere.
m

microscopic-florist-22719

11/03/2018, 12:26 AM
Same for GCP, Azure etc
q

quiet-wolf-18467

11/03/2018, 12:26 AM
Perfect! So the first argument to
createGitLabRunner
could be that provider
m

microscopic-florist-22719

11/03/2018, 12:26 AM
Yes
c

creamy-potato-29402

11/03/2018, 12:26 AM
I thought about just porting the top 20 charts over winter break.
👍 1
g

glamorous-printer-66548

11/03/2018, 12:26 AM
use of static types is so handy for reuse packages.
q

quiet-wolf-18467

11/03/2018, 12:26 AM
Shame there's no pattern matching in TypeScript
c

creamy-potato-29402

11/03/2018, 12:26 AM
lol I know.
seriously.
m

microscopic-florist-22719

11/03/2018, 12:27 AM
@quiet-wolf-18467 if you’re not yet familiar with component resources, you should take a look at them
q

quiet-wolf-18467

11/03/2018, 12:28 AM
I had a quick look, but need to spend more time researching
I didn't think components would be cloud agnostic, but perhaps I didn't play enough
m

microscopic-florist-22719

11/03/2018, 12:29 AM
They can definitely be cloud agnostic. See e.g. cloud.Service, which runs on AWS or Azure
In principle they can even span multiple clouds
q

quiet-wolf-18467

11/03/2018, 12:30 AM
Thanks, @microscopic-florist-22719 👍
m

microscopic-florist-22719

11/03/2018, 12:30 AM
np :)
q

quiet-wolf-18467

11/03/2018, 12:31 AM
Is there a preferred templating system in TypeScript land?
Slight segue
m

microscopic-florist-22719

11/03/2018, 12:31 AM
Beyond interpolated strings?
@white-balloon-205 might know
g

glamorous-printer-66548

11/03/2018, 12:32 AM
I used this for a few things: https://mozilla.github.io/nunjucks/
c

creamy-potato-29402

11/03/2018, 12:33 AM
What do you mean by “templating system”?
g

glamorous-printer-66548

11/03/2018, 12:33 AM
although I prefer to simply use ES6 template literals.
q

quiet-wolf-18467

11/03/2018, 12:38 AM
I'll want to render the GitLab toml from my struct configuration
Into a ConfigMap or user data
I think template literals would make configuring the executor in the toml difficult, @glamorous-printer-66548
g

glamorous-printer-66548

11/03/2018, 12:53 AM
oh yeah toml
toml sucks btw
but
in my package i’m simply using https://www.npmjs.com/package/@iarna/toml
to convert the config from a JS object to a toml file
so no need for template literals or whatever
TOML.stringify(obj)
does the job
q

quiet-wolf-18467

11/03/2018, 12:56 AM
Awesome. Thank you, @glamorous-printer-66548
This may be a silly question ...
When I do
new k8s.apps.v1beta1.Deployment
Does that create the deployment there and then or are all the references collected and run later?
g

glamorous-printer-66548

11/03/2018, 12:58 AM
wdyt of creating a common github organization for some pulumi reuse packages?
q

quiet-wolf-18467

11/03/2018, 12:59 AM
I notice your example, @creamy-potato-29402 uses an annotation to check latency
g

glamorous-printer-66548

11/03/2018, 12:59 AM
something like
pulumi-components
or
pulumi-packages
q

quiet-wolf-18467

11/03/2018, 12:59 AM
But could it just be a normal code block before deploying more?
@glamorous-printer-66548 I created
pulumu-packs
on GitLab
g

glamorous-printer-66548

11/03/2018, 1:00 AM
i noticed but honestly gitlab sucks for open source
their search is so bad you just can’t find anything
q

quiet-wolf-18467

11/03/2018, 1:00 AM
Their search is awful
True
But it's open source and transparent
I'm happy to use GH instead if you think GL would ruin traction
g

glamorous-printer-66548

11/03/2018, 1:01 AM
well it transparently sucks 😄
q

quiet-wolf-18467

11/03/2018, 1:01 AM
We can always mirror to GitHub 😍
"GitHub .. the closed sourced profit sucking company behind the success of open source" 🤣
g

glamorous-printer-66548

11/03/2018, 1:02 AM
yeah i think it would definately hinder traction, it’s just nobody will find it. I bet that gitlab repos also tend to have a lower Google search rank because gitlabs page score is certainly lower
yeah I mean sometimes you just gotta be pragmatic. It’d be more than happy to use gitlab and simply mirror to github, if gitlab would at least fix their search and site speed, but not until then 🙂
q

quiet-wolf-18467

11/03/2018, 1:04 AM
I would use GitHub more if they supported groups. Some orgs have thousands of packages
But this is irrelevant
Why don't you create the org and we can take it from there? 😄
g

glamorous-printer-66548

11/03/2018, 1:05 AM
oh yeah that is interesting
but until we have 1000 pulumi packages there’s a long way 😄
😂 1
yeah sure, but let’s discuss the name first
- pulumi-packs - pulumi-packages - pulumi-components
- … ?
@creamy-potato-29402 any thoughts?
pulumi-charts 😂
😂 1
q

quiet-wolf-18467

11/03/2018, 1:07 AM
Yeah, maybe someone from Pulumi has some input. If we go with pulumi-packs, I'll transfer the npm org 🤗
g

glamorous-printer-66548

11/03/2018, 1:09 AM
ok i’m gonna create pulumi-packs for now. we can rename it a couple of days later if we want to. Pulumi packs is not bad because it’s descriptive (unlike charts) but still distinctive enough to be recognized as own “brand” (compared to “packages”) 🙂
q

quiet-wolf-18467

11/03/2018, 1:10 AM
Sweet
Got the invite. Cheers 👍
g

glamorous-printer-66548

11/03/2018, 1:14 AM
and we have the first repository 🎉 : https://github.com/pulumi-packs/pulumi-packs-typescript
gonna add my gitlab stuff to it in a few mins.
q

quiet-wolf-18467

11/03/2018, 1:16 AM
Would all the typescript packs live in the same repo or should they be separate?
g

glamorous-printer-66548

11/03/2018, 1:17 AM
i would keep them in the same repo for now
planning to start out with a similar structure as the helm chart repo
to have
stable
and
incubator
as top level dirs
we can turn this into seperate repos after some time when there’s enough content but for now it’s easier to keep this as a monorepo for automation etc.
we probably want auto publishing via CI etc.
q

quiet-wolf-18467

11/03/2018, 1:18 AM
Helm have been trying to use separate repos for almost a year and have no idea how to do it
Because it's too late 😃
g

glamorous-printer-66548

11/03/2018, 1:19 AM
lol
ok
hmm
but technically helm is a slightly different story
q

quiet-wolf-18467

11/03/2018, 1:20 AM
One repo is fine, as we have no code. We can change in a few weeks once we understand the scenario more
👍 1
g

glamorous-printer-66548

11/03/2018, 1:20 AM
the problem they’re facing with helm is I believe that there is a single central registry built-in and the only way to publish to that registry is via the single helm/charts repository
with npm instead anybody can publish to the standard npm registry
without having to go through our pulumi packs repo 🙂
q

quiet-wolf-18467

11/03/2018, 1:22 AM
Though if there's a GitLab Runner "pack" and we segregate by language, we can't share config templates across languages
g

glamorous-printer-66548

11/03/2018, 1:23 AM
hmm
q

quiet-wolf-18467

11/03/2018, 1:23 AM
But if the GitLab Runner pack is a repo and there's a directory for each language, we can share templates
g

glamorous-printer-66548

11/03/2018, 1:23 AM
true
but what do you mean as sharing here?
i.e. what parts of the package are “shareable” between languages?
q

quiet-wolf-18467

11/03/2018, 1:24 AM
File based configuration
g

glamorous-printer-66548

11/03/2018, 1:25 AM
hmm
q

quiet-wolf-18467

11/03/2018, 1:25 AM
gitlab.rb
or
runner.toml
, or if it's MySQL then
my.cnf
g

glamorous-printer-66548

11/03/2018, 1:25 AM
I’m not even sure if I want to configure those things via files?
Honestly I personally try to do just everything in typescript
q

quiet-wolf-18467

11/03/2018, 1:26 AM
That won't work for Go and Python
g

glamorous-printer-66548

11/03/2018, 1:26 AM
even config is in typescript files for me lol
We have files like
prod.ts
,
dev.ts
and
config.ts
etc.
well you can configure your app via an object a literal in a
.py file or
.go` file I assume.
anyways
let’s discuss is another time
it’s friday 😛
q

quiet-wolf-18467

11/03/2018, 1:28 AM
It's 2am Saturday 🤣
g

glamorous-printer-66548

11/03/2018, 1:28 AM
I’ll try to spend the next 30 mins on getting my current code up on github and then I gotta find a beer somewhere lol
q

quiet-wolf-18467

11/03/2018, 1:28 AM
Enjoy
Speak later
👍 1
g

glamorous-printer-66548

11/03/2018, 1:29 AM
where are you based?
q

quiet-wolf-18467

11/03/2018, 1:29 AM
Scotland
g

glamorous-printer-66548

11/03/2018, 1:29 AM
whiskey then
😛
c

creamy-potato-29402

11/03/2018, 1:30 AM
@glamorous-printer-66548 @quiet-wolf-18467 sorry… just catching up… you’re making a GH org for pulumi apps?
g

glamorous-printer-66548

11/03/2018, 1:30 AM
yep
c

creamy-potato-29402

11/03/2018, 1:30 AM
I have asked for this internally for my entire tenure here
q

quiet-wolf-18467

11/03/2018, 1:30 AM
For reusable Pulumi libraries
Which we aren't calling charts 😄
c

creamy-potato-29402

11/03/2018, 1:32 AM
honestly? I’m kinda done with nautical analogies.
g

glamorous-printer-66548

11/03/2018, 1:34 AM
@quiet-wolf-18467 fyi I also have some fairly reusable libs for: - https://github.com/kubernetes-incubator/external-dns - https://github.com/jetstack/cert-manager - https://www.reoptimize.io fyi 🙂
c

creamy-potato-29402

11/03/2018, 1:34 AM
cc @white-balloon-205 @big-piano-35669
we really should probably have a place for this kind of thing, at least IMO.
q

quiet-wolf-18467

11/03/2018, 1:35 AM
ExternalDNS and cert manager rely on CRDs though?
You deploy them with Pulumi and kubectl your own manifests?
g

glamorous-printer-66548

11/03/2018, 1:36 AM
external-dns does not rely on CRDs, it relies on annotations
and cert-manager can partially be used also via annotations
c

creamy-potato-29402

11/03/2018, 1:37 AM
Pulumi can handle CRDs, right?
g

glamorous-printer-66548

11/03/2018, 1:38 AM
my packages are also not to “consume” them, they are meant to set them up as cluster-wide services / addons which then can be used by any app running on the cluster. How the app does that is their own choice, if they use kubectl, annotations or CRD objects deployed via kubectl or pulumi, doesn’t matter.
and yes, as hausdorff is right, pulumi supports CRDs.
q

quiet-wolf-18467

11/03/2018, 1:39 AM
ExternalDNS uses CRDs on master. Stop running "releases" 🤣
😄 1
c

creamy-potato-29402

11/03/2018, 1:39 AM
we have some work to make them type safe though.
q

quiet-wolf-18467

11/03/2018, 1:40 AM
CRD maintainers have some work todo also. They don't all provide an OpenAPI spec
c

creamy-potato-29402

11/03/2018, 1:55 AM
that’s right.
how can we build nice tools if they aren’t publishing schemas or OpenAPI specs?
g

glamorous-printer-66548

11/03/2018, 2:10 AM
I think https://github.com/pulumi-packs/pulumi-packs-typescript/blob/master/incubator/gitlab-runner-on-gke/src/gitlab-kubernetes-runner.ts could be turned into a seperate pack which is just responsible for deploying gitlab runner on kubernetes (so it’s GKE independent). The other things are more gke specific in there.