https://pulumi.com logo
Join Slack
Powered by
Is it possible to dynamically configure AWS, GCP, ...
# general
q
Is it possible to dynamically configure AWS, GCP, or K8s?
c
@quiet-wolf-18467 what do you mean when you say “dynamically”?
q
I'm trying to setup "Pulumi Packs" that provide reusable components for people to use
I want to allow passing different GCP or other accounts to the function
c
Hmm, I still am not sure I understand. You mean, passing service accounts around?
maybe it would help if you had an example of the code you’d like to write?
q
Say I want to deploy an pod to 2 different Kubernetes clusters. Is that possible?
c
yeah, sure. In the case of Kubernetes you have two options.
(1) the k8s provider picks up whatever is in your kubeconfig file, so if you can tell the kubernetes provider to use a different context, which points at a different cluster. For this to be sensible you’d need one stack per cluster, though.
q
And if I wanted it in a single stack?
c
(2) in the case of a high-order provider, you’d just specify two different providers. So say you boot up an AKS and GKE cluster in the same app. You’d write a 
for
loop over both.
q
So providers can be configured in code? That's what I'm after. Can the same be done for AWS, Azure, and GCS?
GCP, sorry
c
hmm, you’d probably want to have 2 providers, one for each cluster.
yeah.
That I do not know.
q
c
@microscopic-florist-22719?
q
It's super early. But I want to provide something that is configurable like that
c
I’d love to help.
Pat is the guy to ask though.
q
I plan to provide for TypeScript, Python, and Go
And I want to write functions for Kubernetes deployments to check InfluxDB and Prometheus for linear regression and automated rollbscks
So it's all plug and play
c
oooo have you seen our staged rollout example for k8s?
if not, check it out.
g
@quiet-wolf-18467 funny to see that. I actually built a pulumi package to setup gitlab runner as well. In my case it’s setting up a gke cluster from scratch and installs an auto scaling gitlab-runner onto it. Maybe I should open source it - it’s pretty generic already.
q
I've seen the stages roll out. But it's not that reusable.
c
that’s true.
you mean in the sense that it’s not built like a library?
q
Exactly
c
maybe I should make it a library
q
I'd like to just import the function 😄
c
yeah
it was meant as an example, but maybe it’s worth doing as a library
q
@glamorous-printer-66548 I'd love to see it
I think there's many thing that can be shared this way. I think Pulumi can replace Helm. While I know Pulumi can deploy Helm charts, it's also much better to codify than Go Templates
👍 1
I hope I'm thinking things that are already on the teams minds and not going too far down the crazy path 😃
m
Late to the party here, but yes you can do this.
With one catch: neither Python nor Go expose first-class providers yet.
c
@quiet-wolf-18467 well, on the k8s side, my goal is to make this situation really easy, yes.
m
In TypeScript, you just do 
new aws.Provider(“name”, { inputs })
like with any other resource
g
@quiet-wolf-18467 absolutely agree. Would love to have a helm-like ecosystem for pulumi packages 🙂
c
i.e. to make it really easy to author libraries you can use anywhere.
m
Same for GCP, Azure etc
q
Perfect! So the first argument to 
createGitLabRunner
could be that provider
m
Yes
c
I thought about just porting the top 20 charts over winter break.
👍 1
g
use of static types is so handy for reuse packages.
q
Shame there's no pattern matching in TypeScript
c
lol I know.
seriously.
m
@quiet-wolf-18467 if you’re not yet familiar with component resources, you should take a look at them
q
I had a quick look, but need to spend more time researching
I didn't think components would be cloud agnostic, but perhaps I didn't play enough
m
They can definitely be cloud agnostic. See e.g. cloud.Service, which runs on AWS or Azure
In principle they can even span multiple clouds
q
Thanks, @microscopic-florist-22719 👍
m
np :)
q
Is there a preferred templating system in TypeScript land?
Slight segue
m
Beyond interpolated strings?
@white-balloon-205 might know
g
I used this for a few things: https://mozilla.github.io/nunjucks/
c
What do you mean by “templating system”?
g
although I prefer to simply use ES6 template literals.
q
I'll want to render the GitLab toml from my struct configuration
Into a ConfigMap or user data
I think template literals would make configuring the executor in the toml difficult, @glamorous-printer-66548
g
oh yeah toml
toml sucks btw
but
in my package i’m simply using https://www.npmjs.com/package/@iarna/toml
to convert the config from a JS object to a toml file
so no need for template literals or whatever
TOML.stringify(obj)
does the job
q
Awesome. Thank you, @glamorous-printer-66548
This may be a silly question ...
When I do 
new k8s.apps.v1beta1.Deployment
Does that create the deployment there and then or are all the references collected and run later?
g
wdyt of creating a common github organization for some pulumi reuse packages?
q
I notice your example, @creamy-potato-29402 uses an annotation to check latency
g
something like 
pulumi-components
or 
pulumi-packages
…
q
But could it just be a normal code block before deploying more?
@glamorous-printer-66548 I created 
pulumu-packs
on GitLab
g
i noticed but honestly gitlab sucks for open source
their search is so bad you just can’t find anything
q
Their search is awful
True
But it's open source and transparent
I'm happy to use GH instead if you think GL would ruin traction
g
well it transparently sucks 😄
q
We can always mirror to GitHub 😍
"GitHub .. the closed sourced profit sucking company behind the success of open source" 🤣
g
yeah i think it would definately hinder traction, it’s just nobody will find it. I bet that gitlab repos also tend to have a lower Google search rank because gitlabs page score is certainly lower
yeah I mean sometimes you just gotta be pragmatic. It’d be more than happy to use gitlab and simply mirror to github, if gitlab would at least fix their search and site speed, but not until then 🙂
q
I would use GitHub more if they supported groups. Some orgs have thousands of packages
But this is irrelevant
Why don't you create the org and we can take it from there? 😄
g
oh yeah that is interesting
but until we have 1000 pulumi packages there’s a long way 😄
😂 1
yeah sure, but let’s discuss the name first
- pulumi-packs - pulumi-packages - pulumi-components
- … ?
@creamy-potato-29402 any thoughts?
pulumi-charts 😂
😂 1
q
Yeah, maybe someone from Pulumi has some input. If we go with pulumi-packs, I'll transfer the npm org 🤗
g
ok i’m gonna create pulumi-packs for now. we can rename it a couple of days later if we want to. Pulumi packs is not bad because it’s descriptive (unlike charts) but still distinctive enough to be recognized as own “brand” (compared to “packages”) 🙂
q
Sweet
Got the invite. Cheers 👍
g
and we have the first repository 🎉 : https://github.com/pulumi-packs/pulumi-packs-typescript
gonna add my gitlab stuff to it in a few mins.
q
Would all the typescript packs live in the same repo or should they be separate?
g
i would keep them in the same repo for now
planning to start out with a similar structure as the helm chart repo
to have 
stable
and 
incubator
as top level dirs
we can turn this into seperate repos after some time when there’s enough content but for now it’s easier to keep this as a monorepo for automation etc.
we probably want auto publishing via CI etc.
q
Helm have been trying to use separate repos for almost a year and have no idea how to do it
Because it's too late 😃
g
lol
ok
hmm
but technically helm is a slightly different story
q
One repo is fine, as we have no code. We can change in a few weeks once we understand the scenario more
👍 1
g
the problem they’re facing with helm is I believe that there is a single central registry built-in and the only way to publish to that registry is via the single helm/charts repository
with npm instead anybody can publish to the standard npm registry
without having to go through our pulumi packs repo 🙂
q
Though if there's a GitLab Runner "pack" and we segregate by language, we can't share config templates across languages
g
hmm
q
But if the GitLab Runner pack is a repo and there's a directory for each language, we can share templates
g
true
but what do you mean as sharing here?
i.e. what parts of the package are “shareable” between languages?
q
File based configuration
g
hmm
q
gitlab.rb
or 
runner.toml
, or if it's MySQL then 
my.cnf
g
I’m not even sure if I want to configure those things via files?
Honestly I personally try to do just everything in typescript
q
That won't work for Go and Python
g
even config is in typescript files for me lol
We have files like 
prod.ts
, 
dev.ts
and 
config.ts
etc.
well you can configure your app via an object a literal in a 
.py file or
.go` file I assume.
anyways
let’s discuss is another time
it’s friday 😛
q
It's 2am Saturday 🤣
g
I’ll try to spend the next 30 mins on getting my current code up on github and then I gotta find a beer somewhere lol
q
Enjoy
Speak later
👍 1
g
where are you based?
q
Scotland
g
whiskey then
😛
c
@glamorous-printer-66548 @quiet-wolf-18467 sorry… just catching up… you’re making a GH org for pulumi apps?
g
yep
c
I have asked for this internally for my entire tenure here
q
For reusable Pulumi libraries
Which we aren't calling charts 😄
c
honestly? I’m kinda done with nautical analogies.
g
@quiet-wolf-18467 fyi I also have some fairly reusable libs for: - https://github.com/kubernetes-incubator/external-dns - https://github.com/jetstack/cert-manager - https://www.reoptimize.io fyi 🙂
c
cc @white-balloon-205 @big-piano-35669
we really should probably have a place for this kind of thing, at least IMO.
q
ExternalDNS and cert manager rely on CRDs though?
You deploy them with Pulumi and kubectl your own manifests?
g
external-dns does not rely on CRDs, it relies on annotations
and cert-manager can partially be used also via annotations
c
Pulumi can handle CRDs, right?
g
my packages are also not to “consume” them, they are meant to set them up as cluster-wide services / addons which then can be used by any app running on the cluster. How the app does that is their own choice, if they use kubectl, annotations or CRD objects deployed via kubectl or pulumi, doesn’t matter.
and yes, as hausdorff is right, pulumi supports CRDs.
q
ExternalDNS uses CRDs on master. Stop running "releases" 🤣
😄 1
c
we have some work to make them type safe though.
q
CRD maintainers have some work todo also. They don't all provide an OpenAPI spec
c
that’s right.
how can we build nice tools if they aren’t publishing schemas or OpenAPI specs?
g
I think https://github.com/pulumi-packs/pulumi-packs-typescript/blob/master/incubator/gitlab-runner-on-gke/src/gitlab-kubernetes-runner.ts could be turned into a seperate pack which is just responsible for deploying gitlab runner on kubernetes (so it’s GKE independent). The other things are more gke specific in there.
Open in Slack
PreviousNext
Powered by