Off-topic: Using pulumi sparked an idea in me: How...
# general
Off-topic: Using pulumi sparked an idea in me: How about generating YAML files for YAML-based CI systems (circle, travis, gitlab and basically almost every “modern” ci system) via a CLI and a bunch of TypeScript libraries? Why? We’re using more and more of gitlab-ci and either our config files becomes super large or we’re using a lot of yaml anchors which are not exactly readable and it’s a lot of try and error to get actually valid YAML out of it. In general it’s also often not clear what config options an CI system offers without checking the documentation in detail. I think it should be possible (and probably not super hard) to build a TypeScript library which can generate for example gitlab CI yamls and is strictly typed, so if you put in some invalid config keys / values the compiler yells at you instead of having to run a lot of try-and-error ci jobs. Also it would be possible to build some reuse / abstraction library to create a set of common jobs that can be reused in multiple repos. Wdyt think of something like this? Anybody heard whether something similar already exists?
Those YAML files needs to exist on the
git push
, so you'd use Pulumi to generate them and auto commit? I generally think auto-commits are an anti-pattern
Well, maybe this wasn’t clear: I would not use pulumi for this at all. Pulumi’s purpose is different, but the switch from k8s yamls to using TypeScript made me just thinking how I can get rid of those CI yaml mess.
I think I would generate them in a commit post-hook or simply run the generation manually via
npm run
, some shell script or some CLI ….. Yeah it’s nasty having to run some “generate” step after every change, but I think I prefer this tradeoff over yaml-try-and-error and not having the ability to reuse things easily.
Running the “generate” step serves basically as a more powerful replacement for some ci yaml validator.
I use GitLab includes
I’m currently enabling pulumi deployment via gitlab in some of our repos, but I’m just repeating myself too much / copy and paste a lot, even if making using of yaml anchors etc. a lot.
but there's also some nice systems that use Kubernetes CRDs for build pipelines
Such as InfraBox, Knative, and Avro
Oh, and I shouldn't forget Brigade, which lets you code build pipelines in JS
yeah i’m aware of gitlab includes, but they don’t support yaml anchors in the include file, so I don’t think this will help me to the extent I want it to.
infrabox uses JSON config, how’s that better?
I used to work for SAP btw ^^
yeah brigade is a nice “idea” but I don’t think it’s nicely executed unfortunately… I tried it a couple of months ago but was very confused by how it works
The CI / Automation systems I know of which support some higher level language are: Jenkins (Groovy DSL), Teamcity (Kotlin Script), buildbot (python), Jenkins X, Brigade. Unfortunately each of them is fully self-hosted. It’d rather use one of those mature managed, docker oriented services like circle and gitlab 😕
ok for gitlab there’s an open issue to allow dynamic config file generation and a few folks in that issue have apparently implemented some config file generation already:
just not in typescript yet
Oh, I thought InfraBox was CRDs now. I was incorrect 🙂
Argo is CRDs
option would be awesome
Argo, that's it's name! I think I called it Avro 😂
I keep looking at it and watching YouTube demonstrations, but I haven't moved away from GitLab CI yet
I very keen to separate CI from CD
CI - Build OCI Image CD - Deploy with Pulumi
That's what I want
Avro is a file format 😛
By chance we actually use both Argo and Avro in my company
Argo is actually pretty nice, I think eventually I might just replace gitlab ci with it haha. It’s just that it doesn’t really have good UI integration to gitlab or github…
Currently we use argo for some machine learning pipelines which is slightly different use case to CI / CD, but there are intersections.
oh I forgot, Airflow (python) technically also could be used for CI/CD, but just like argo its more build for data pipelines than CI/CD.
Anyways, I gotta go. I think I might build some MVC for generating gitlab-ci yamls with typescript in the coming days.
can’t be that hard.
Best of luck 👍
I have an example of this for circle ci. Nothing that sources it though so our devs are forced to setup pre-commit hooks