Off-topic: Using pulumi sparked an idea in me:
How...
# generalg
glamorous-printer-66548
10/12/2018, 9:28 AMOff-topic: Using pulumi sparked an idea in me:
How about generating YAML files for YAML-based CI systems (circle, travis, gitlab and basically almost every “modern” ci system) via a CLI and a bunch of TypeScript libraries? Why?
We’re using more and more of gitlab-ci and either our config files becomes super large or we’re using a lot of yaml anchors which are not exactly readable and it’s a lot of try and error to get actually valid YAML out of it. In general it’s also often not clear what config options an CI system offers without checking the documentation in detail. I think it should be possible (and probably not super hard) to build a TypeScript library which can generate for example gitlab CI yamls and is strictly typed, so if you put in some invalid config keys / values the compiler yells at you instead of having to run a lot of try-and-error ci jobs. Also it would be possible to build some reuse / abstraction library to create a set of common jobs that can be reused in multiple repos.
Wdyt think of something like this? Anybody heard whether something similar already exists?
q
quiet-wolf-18467
10/12/2018, 9:32 AM
Those YAML files needs to exist on the
git pushg
glamorous-printer-66548
10/12/2018, 9:33 AMWell, maybe this wasn’t clear: I would not use pulumi for this at all. Pulumi’s purpose is different, but the switch from k8s yamls to using TypeScript made me just thinking how I can get rid of those CI yaml mess.
glamorous-printer-66548
10/12/2018, 9:36 AMI think I would generate them in a commit post-hook or simply run the generation manually via
npm runglamorous-printer-66548
10/12/2018, 9:38 AMRunning the “generate” step serves basically as a more powerful replacement for some ci yaml validator.
q
quiet-wolf-18467
10/12/2018, 9:41 AM
quiet-wolf-18467
10/12/2018, 9:41 AM
I use GitLab includes
g
glamorous-printer-66548
10/12/2018, 9:41 AMI’m currently enabling pulumi deployment via gitlab in some of our repos, but I’m just repeating myself too much / copy and paste a lot, even if making using of yaml anchors etc. a lot.
q
quiet-wolf-18467
10/12/2018, 9:41 AM
but there's also some nice systems that use Kubernetes CRDs for build pipelines
quiet-wolf-18467
10/12/2018, 9:42 AM
Such as InfraBox, Knative, and Avro
quiet-wolf-18467
10/12/2018, 9:42 AM
Oh, and I shouldn't forget Brigade, which lets you code build pipelines in JS
g
glamorous-printer-66548
10/12/2018, 9:42 AMyeah i’m aware of gitlab includes, but they don’t support yaml anchors in the include file, so I don’t think this will help me to the extent I want it to.
glamorous-printer-66548
10/12/2018, 9:43 AMinfrabox uses JSON config, how’s that better?
glamorous-printer-66548
10/12/2018, 9:43 AMI used to work for SAP btw ^^
glamorous-printer-66548
10/12/2018, 9:45 AMyeah brigade is a nice “idea” but I don’t think it’s nicely executed unfortunately… I tried it a couple of months ago but was very confused by how it works
glamorous-printer-66548
10/12/2018, 9:49 AMThe CI / Automation systems I know of which support some higher level language are: Jenkins (Groovy DSL), Teamcity (Kotlin Script), buildbot (python), Jenkins X, Brigade. Unfortunately each of them is fully self-hosted. It’d rather use one of those mature managed, docker oriented services like circle and gitlab 😕
glamorous-printer-66548
10/12/2018, 9:56 AMglamorous-printer-66548
10/12/2018, 10:08 AMok for gitlab there’s an open issue to allow dynamic config file generation and a few folks in that issue have apparently implemented some config file generation already: https://gitlab.com/gitlab-org/gitlab-ce/issues/45828
glamorous-printer-66548
10/12/2018, 10:08 AMjust not in typescript yet
q
quiet-wolf-18467
10/12/2018, 10:08 AM
Oh, I thought InfraBox was CRDs now. I was incorrect 🙂
g
glamorous-printer-66548
10/12/2018, 10:09 AMArgo is CRDs
q
quiet-wolf-18467
10/12/2018, 10:09 AM
Oh
quiet-wolf-18467
10/12/2018, 10:09 AM
That
seedquiet-wolf-18467
10/12/2018, 10:10 AM
Argo, that's it's name! I think I called it Avro 😂
quiet-wolf-18467
10/12/2018, 10:10 AM
I keep looking at it and watching YouTube demonstrations, but I haven't moved away from GitLab CI yet
quiet-wolf-18467
10/12/2018, 10:10 AM
I very keen to separate CI from CD
quiet-wolf-18467
10/12/2018, 10:10 AM
CI - Build OCI Image
CD - Deploy with Pulumi
quiet-wolf-18467
10/12/2018, 10:10 AM
That's what I want
g
glamorous-printer-66548
10/12/2018, 10:11 AMAvro is a file format 😛
glamorous-printer-66548
10/12/2018, 10:11 AMBy chance we actually use both Argo and Avro in my company
glamorous-printer-66548
10/12/2018, 10:11 AMArgo is actually pretty nice, I think eventually I might just replace gitlab ci with it haha. It’s just that it doesn’t really have good UI integration to gitlab or github…
glamorous-printer-66548
10/12/2018, 10:12 AMCurrently we use argo for some machine learning pipelines which is slightly different use case to CI / CD, but there are intersections.
glamorous-printer-66548
10/12/2018, 10:13 AMoh I forgot, Airflow (python) technically also could be used for CI/CD, but just like argo its more build for data pipelines than CI/CD.
glamorous-printer-66548
10/12/2018, 10:15 AMAnyways, I gotta go. I think I might build some MVC for generating gitlab-ci yamls with typescript in the coming days.
glamorous-printer-66548
10/12/2018, 10:15 AMcan’t be that hard.
q
quiet-wolf-18467
10/12/2018, 10:16 AM
Best of luck 👍
b
busy-umbrella-36067
10/12/2018, 1:35 PMI have an example of this for circle ci. Nothing that sources it though so our devs are forced to setup pre-commit hooks
3 Views