i'm starting to think about ci/cd with pulumi. I've got a monorepo of pulumi projects with multiple stacks for most projects. I'm thinking I need a tool to figure out which projects have been changed, run preview/up on each one of those projects and also a way of specifying the order in which the preview/ups are applied. Are there any existing open source tools that do this? Would love to hear from others who have gone down this path and how they've done it.

Depending on which CI/CD platform you are using you should be able to create triggers on when pipelines are completed.

hey! which CI/CD tool are you planning to use?

@calm-parrot-72437 We’re in a similar situation. We developed a basic tool that compares all the changes from the last successful build and then executes them all in their alphabetical order. It’s not necessarily the nicest code, but it works. It’s not open source at the moment, but it could be.

I'll be using gitlab

We haven’t run into such an issue at the moment. We’d be happy to accept a PR if we open sourced it to add such support.

do you keep your commits to one project at at time? @cool-egg-852

Not generally. But we also don’t generally make changes that depend upon each other.

sometimes i wonder if I'm doing this right 😉

I’d be happy to discuss our architecture a bit with you on a private call. If you are interested, DM me.

i'm on kubernetes and i'll keep the Deployment/service, etc in one project. but if hte project relies on other infrastructure, say s3 buckets, a filesystem, database, etc., those will be done elsewhere

That should all be in one project generally.

well, they might be shared between projects, so which one would it be deployed in

We always pick an “owner” application.

but i'd think you'd still have dependencies, to deploy my kubenetes applications i'm reliant on the outputs from the project that brings up the cluster, for example. this doesn't change, but being able to model the dependencies appropriately means i could bring up a new environment quickly. without doing, i'd still have manual work to do.

We separate kubernetes itself into a separate project.

exactly and those become dependencies of your applications. you have to control the order of operations there, if perhaps only at the very beginning as the environments are created

But outside of these few specific changes, we don’t have a ton of dependencies. Our big one is just the API hostname that is used by other services. However that never changes, so we do this a little manually when we have a new service.

yeah, maybe i don't sweat the small stuff and leave the beginning mostly manual

That’s how we do it. I recommend documenting it, so if you do need to go through the process again, you have the order there.

yeah, i only have one working environment now and thinking about how to go to 1 more and then maintain consistency there.

We do a staging + production setup, both of which are nearly identical, with whatever changes being handle via configuration.

i see. yeah, i was thinking about using different branches.. but that doesn't map perfectly to how i'm doing things right now

For example, we’re switching away from istio to linkerd, so we have a

project:useLinkerd

bool in the

Pulumi.staging.yaml

file. If it’s true, we use the linkerd configuration, otherwise we use the

istio

configuration. This to us is easier to manage than separate branches.

I’m a fan of ConcourseCI. Their

git

support allows for a

path

specification which results in only triggering builds when files within that(/those) path(s) are changed. https://concourse-ci.org/ https://github.com/concourse/git-resource It is also the only CI where pipelines can be modelled over multiple git repositories. See Concourse CI building its own codebase as an example: https://ci.concourse-ci.org/teams/main/pipelines/concourse

oh that is nice, Ringo. i've always had to build that in the past myself.