Off topic Using pulumi sparked an idea in me How about gener

Question

Off topic Using pulumi sparked an idea in me How about generating YAML files for YAML based CI systems circle travis gitlab and basically almost every modern ci system via a CLI and a bunch of TypeScript libraries Why We re using more and more of gitlab ci and either our config files becomes super large or we re using a lot of yaml anchors which are not exactly readable and it s a lot of try and error to get actually valid YAML out of it In general it s also often not clear what config options an CI system offers without checking the documentation in detail I think it should be possible and probably not super hard to build a TypeScript library which can generate for example gitlab CI yamls and is strictly typed so if you put in some invalid config keys values the compiler yells at you instead of having to run a lot of try and error ci jobs Also it would be possible to build some reuse abstraction library to create a set of common jobs that can be reused in multiple repos Wdyt think of something like this Anybody heard whether something similar already exists

quiet-wolf-18467 · Answer

Those YAML files needs to exist on the `git push` so you d use Pulumi to generate them and auto commit I generally think auto commits are an anti pattern

glamorous-printer-66548 · Answer

Well maybe this wasn t clear I would not use pulumi for this at all Pulumi s purpose is different but the switch from k8s yamls to using TypeScript made me just thinking how I can get rid of those CI yaml mess

glamorous-printer-66548 · Answer

I think I would generate them in a commit post hook or simply run the generation manually via `npm run` some shell script or some CLI Yeah it s nasty having to run some generate step after every change but I think I prefer this tradeoff over yaml try and error and not having the ability to reuse things easily

glamorous-printer-66548 · Answer

Running the generate step serves basically as a more powerful replacement for some ci yaml validator

quiet-wolf-18467 · Answer

quiet-wolf-18467 · Answer

I use GitLab includes

glamorous-printer-66548 · Answer

I m currently enabling pulumi deployment via gitlab in some of our repos but I m just repeating myself too much copy and paste a lot even if making using of yaml anchors etc a lot

quiet-wolf-18467 · Answer

but there s also some nice systems that use Kubernetes CRDs for build pipelines

quiet-wolf-18467 · Answer

Such as InfraBox Knative and Avro

quiet-wolf-18467 · Answer

Oh and I shouldn t forget Brigade which lets you code build pipelines in JS

glamorous-printer-66548 · Answer

yeah i m aware of gitlab includes but they don t support yaml anchors in the include file so I don t think this will help me to the extent I want it to

glamorous-printer-66548 · Answer

infrabox uses JSON config how s that better

glamorous-printer-66548 · Answer

I used to work for SAP btw ^^

glamorous-printer-66548 · Answer

yeah brigade is a nice idea but I don t think it s nicely executed unfortunately I tried it a couple of months ago but was very confused by how it works

glamorous-printer-66548 · Answer

The CI Automation systems I know of which support some higher level language are Jenkins Groovy DSL Teamcity Kotlin Script buildbot python Jenkins X Brigade Unfortunately each of them is fully self hosted It d rather use one of those mature managed docker oriented services like circle and gitlab confused

glamorous-printer-66548 · Answer

<https gitlab com gitlab org gitlab ce blob master gitlab ci yml> face vomiting

glamorous-printer-66548 · Answer

ok for gitlab there s an open issue to allow dynamic config file generation and a few folks in that issue have apparently implemented some config file generation already <https gitlab com gitlab org gitlab ce issues 45828>

glamorous-printer-66548 · Answer

just not in typescript yet

quiet-wolf-18467 · Answer

Oh I thought InfraBox was CRDs now I was incorrect slightly smiling face

glamorous-printer-66548 · Answer

Argo is CRDs

quiet-wolf-18467 · Answer

Oh

quiet-wolf-18467 · Answer

That `seed` option would be awesome

quiet-wolf-18467 · Answer

Argo that s it s name I think I called it Avro joy

quiet-wolf-18467 · Answer

I keep looking at it and watching YouTube demonstrations but I haven t moved away from GitLab CI yet

quiet-wolf-18467 · Answer

I very keen to separate CI from CD

quiet-wolf-18467 · Answer

CI Build OCI Image CD Deploy with Pulumi

quiet-wolf-18467 · Answer

That s what I want

glamorous-printer-66548 · Answer

Avro is a file format stuck out tongue

glamorous-printer-66548 · Answer

By chance we actually use both Argo and Avro in my company

glamorous-printer-66548 · Answer

Argo is actually pretty nice I think eventually I might just replace gitlab ci with it haha It s just that it doesn t really have good UI integration to gitlab or github

glamorous-printer-66548 · Answer

Currently we use argo for some machine learning pipelines which is slightly different use case to CI CD but there are intersections

glamorous-printer-66548 · Answer

oh I forgot Airflow python technically also could be used for CI CD but just like argo its more build for data pipelines than CI CD

glamorous-printer-66548 · Answer

Anyways I gotta go I think I might build some MVC for generating gitlab ci yamls with typescript in the coming days

glamorous-printer-66548 · Answer

can t be that hard

quiet-wolf-18467 · Answer

Best of luck +1

busy-umbrella-36067 · Answer

I have an example of this for circle ci Nothing that sources it though so our devs are forced to setup pre commit hooks