https://pulumi.com logo
#general
Title
# general
f

freezing-umbrella-5483

05/06/2022, 11:48 PM
Is it possible to provision a CI/CD pipeline from within the Pulumi code, like it's done in AWS CDK Pipelines?
v

victorious-church-57397

05/07/2022, 12:08 PM
We use pulumi in conjunction with GitHub actions. You could use Pulumi to configure GitHub repositories, then you could use GH actions to execute your Pulumi
f

freezing-umbrella-5483

05/07/2022, 12:21 PM
No. CDK pipelines make it easy and abstract to deploy self mutating pipelines that deploy cdk stacks to different environment.
They're built on top of CodePipeline.
v

victorious-church-57397

05/07/2022, 12:24 PM
ah ok, ive never used cdk myself but at pulumi up they announced the interop between pulumi and cdk, you might find something useful in here: https://github.com/pulumi/pulumi-cdk
f

freezing-umbrella-5483

05/07/2022, 12:32 PM
I've asked whether CDK pipelines were supported over at #pulumi-cdk, but didn't get a response. I don't think it can be supported, though, that's why I asked here for an alternative.
Does anyone have any experience or ideas on how to achieve this?
g

great-sunset-355

05/10/2022, 5:12 PM
I'll repeat myself. If you can AVOID all AWS Code** solutions because they are not developer friendly at all. You'll get done everything 3x faster with other CI solutions even Jenkins
You can ofc provision CodePipeline from pulumi but there are so many small pieces and if you do not understand all of them it's pretty daunting experience.
f

freezing-umbrella-5483

05/10/2022, 5:14 PM
I've been using CodePipeline/CodeBuild for a while and they work fine. I'm not married to them, that's why my original question doesn't mention them. I'm asking if it's possible to provision a self-mutating CI/CD process in the same Pulumi project that deploys the actual apps.
Yeah, CDK pipelines abstracts all of those small pieces away, that's why it's a great experience. I just mentioned it as an example of what I want to achieve.
g

great-sunset-355

05/10/2022, 5:16 PM
That's the beauty of all other CI/CD pipelines, you do not need anything self mutating garbage. But you can achieve that with pulumi I'm sure.
f

freezing-umbrella-5483

05/10/2022, 5:17 PM
Why don't I? For example, if I want to add another stage in the pipeline, I want to be able to do it with a simple
git push
of my IaC code. The pipeline fetches the code and updates itself.
Can you give me any pointers on how this can be achieved with Pulumi?
g

great-sunset-355

05/10/2022, 5:20 PM
because pipeline definition lives together with code.. eg GitlabCI - just add 1 section in yaml and that's it. For implementation, you'll need to write quite a bit of code and end up with a similar code CDK provides
f

freezing-umbrella-5483

05/10/2022, 5:22 PM
I don't want to write yaml, I want to write my IaC in TS/Python/etc, because the CI/CD definition is just as dynamic as the other infra.
g

great-sunset-355

05/10/2022, 5:22 PM
the point of gitlab/github is that you do not care about infra at all
the complexity fo AWS Code** solutions and their deployment is the main reason why I recommend against that unless there is a reason. eg. regulations ... it's almost faster to deploy the entire self-hosted Gitlab and configure runners than writing magical CodePipeline
f

freezing-umbrella-5483

05/10/2022, 5:24 PM
CI/CD is infra too. As mentioned, the pipeline definitions are dynamic and depend on the rest of the infra code. For example, I want to be able to automatically provision/tear down pipelines based on my infra code, and to modify them as well. Does this have to involve manual changes to the yaml config, or can Pulumi handle this?
g

great-sunset-355

05/10/2022, 5:29 PM
pulumi gives you full programming language to do it. I'm just saying it's not worth it. I have pipelines with 10+ stages/jobs in Github/Gitlab and if I need to disable anything it's as simple as commenting out a few lines of YAML file. And I did not have to touch any infra at all. The experience you are trying to achieve with CodePipeline is exactly that but at the cost of your time developing workarounds for crippled AWS Service.
you can read CDK Pipelines code to get the idea what would you need to do with Pulumi
f

freezing-umbrella-5483

05/10/2022, 6:20 PM
if I need to disable anything it's as simple as commenting out a few lines of YAML file.
You're missing the point - I'm saying I want to change a line in my Pulumi code, and that should not only mutate the infra, but the pipelines as well - for example, provision new ones. I don't want to make any manual changes to yaml.
you can read CDK Pipelines code to get the idea what would you need to do with Pulumi
My issue is that I don't have much experience with Pulumi, so I wouldn't know what it would take to achieve it. I was hoping this was a solved problem, that's why I asked here.
g

great-sunset-355

05/10/2022, 6:31 PM
Here are very few users of AWS Code* tools, usually, the main reason to use it is silly compliance. So the chance this is solved is very low. Writing Pulumi is rather simple however you need application architecture skills. For example to add actions to the pipeline, you can read the directory that contains your projects.
Copy code
infra/
app/1
app/2
so every time you add something to
app/*
the infra code will look there and creates/removes action/job/stage - but I think with CodePipeline it's going to be a lot of code
s

stocky-restaurant-98004

05/10/2022, 11:26 PM
@freezing-umbrella-5483 Hi. I'm a Pulumi employee and have also used CDK Pipelines in a previous job as a consultant. I think the short answer is "no". Right now you'd have to create this from the raw resources in the AWS provider. I'm aware firsthand of how difficult this is. I would probably try creating the pipeline with regular CDK constructs and using the CDK-Pulumi integration to deploy. While they aren't self-mutating and it's slightly more code compared to CDK Pipelines, they will take care of what can be some very tricky IAM perms. I've created a feature request in AWSX. Feel free to upvote if you would prefer a native Pulumi component (although it would not be self-mutating): https://github.com/pulumi/pulumi-awsx/issues/835
f

freezing-umbrella-5483

05/11/2022, 9:44 AM
@stocky-restaurant-98004 thank you for the response! Unfortunately, it's far from "slightly" more code, as CDK pipelines abstracts so much away - especially with deploying to multiple environments (accounts/regions) and with creating actions for building/uploading assets automatically. I'm not married to AWS's CI/CD tools, as mentioned, so I'd welcome an abstraction like CDK Pipelines that works with other providers as well. The ultimate goal is to have
pulumi up
provision CI/CD pipelines that will in turn use
pulumi up
to deploy the app itself, and have it all dynamic and within a single app. Not sure I explained it well, but you know what I'm talking about, having worked with CDK pipelines yourself. Appreciate the feature request!
6 Views