Hey all, new to pulumi and also this side of azure — hope asking here is ok. I have a greenfield project at hand and want to implement rigorous continues delivery principles. I’m however new to infrastructure as code (but absolutely love the idea). • What I am struggling with, is how to deal with the data side of things (replicating data into staging, backups, migration). • In simple terms, I struggle to wrap my head around my (naive) idea: ◦ create a copy of the production system incl. data, users etc as “test group” ▪︎ apply database migrations / schema changes ▪︎ have tests run in that resource group (which changes data, adds, removes records) and ▪︎ if everything works fine, delete that “testing group” ◦ create a new resource group that again copies from production but only applies schema changes ▪︎ gets promoted to the new production system ◦ the old production system gets destroyed The stack is “simple”: Github Actions, Azure PostGres SQL, Codebase pure TypeScript / Node.js / React, Services are maybe Containers with Red Panda (kafka) and Materialize I’d like to implement this in a deterministic fashion with pulumi 😒weaty-ham: … but really have never pushed CI/CD through to infrastructure before. thank you very much

One could argue we past infrastructure as code (IaC) and into general DevOps. This request more than just IaC with pulumi. For creating the “test group”, you can specify the create mode to be a restore. Then use your existing templates to create the Azure resources. This would get you a copy of the production server. Recommend finding a way to scrub the data too because might hit compliance issues pulling production data some where else. For applying schema changes and all PostgreSQL manipulation that would be psql. You could then wrap the psql with pulumi command. This in theory makes pulumi aware of the psql. If everything good, see if the Azure PostgreSQL action can deploy the changes.

Well the production data would stay within the same org of course. Thank you for pointing me to the Azure Postgres SQL action. Do you know of “standard ways” how to do data migrations in azure even without pulumi? In general, how to deal with that challange from a devops perspective? Or rather: How would you do it?

How you doing your migrations now? We use ef core for our C# projects and just run ef core commands in our CI/CD pipeline. Look into running your migration commands as a shell task or a third party tool.

Well schema migrations are done as part of prisma — pretty much an ORM for typescript. In the past those migrations have been done half manually where I was not in charge. Meaning: someone did a back-up than triggered the migration and checked if everything is kinda working. Doing migrations directly against the production database from CI/CD of course works — but I don‘t fully trust the automatic migration part and would like to run tests in the production environment against that system. Hence the idea to replicate everything; be fine to do everything automatically and have tests check if you broke something. I guess you „simply“ trigger a backup of the database and run the migration (when all tests pass) as part of the CI process? Maybe I should trust the migrations more. Mh.

Deploy an Azure function with HTTP trigger as part of the “test group.” The pulumi template can output the URL with key. Then have a GitHub action call the function. The function would run queries to ensure the new schema is working. If no errors return back then golden. If there are errors then the pipeline fails and go troubleshoot. If all passes then move onto other testing or move to production deployment. It funny you mentioned Prisma, it something I need to do more research on. Developers having some pain on another project and that one of the tools I stumbled across.

Haha yeah prisma — the guys here in berlin are practically my naighbours. Almost joined them back in 2018 — they are a great GraphQL-Database ORM. However, I personally try to get away from CRUD to event sourcing. And events are not really the thing you want to do on a database schema but rather a Kafka schema — that‘s the reason I didn‘t use them much in the past but in this project here, event sourcing would be overkill for an MVP stage. So crud it is… 😪

Hey Dino - we're embarking on the same thing - context is: apps are being deployed into a k8s cluster in Azure. How did you get on?