https://pulumi.com logo
Join Slack
Powered by
Is there anyone that uses self-managed backend(aws...
# general
p
Is there anyone that uses self-managed backend(aws/gcp/azure) and implemented its own concurrency, rollback mechanism etc
v
hi, by concurrency, do you mean locking down how many operations you can have on resources? pulumi will lock a stack once a 
pulumi up
has started, so no other invocations of 
pulumi up
can run against that stack. as for rollback - you could write a wrapper script around 
pulumi up
so that it snapshots the state, executes 
pulumi up
, and if it fails, re-imports the state. you'd then need to execute 
pulumi up
again either in the script or manually to ensure the resources are configured in their previous state
p
Actually, taken https://www.pulumi.com/docs/iac/concepts/state-and-backends I'm not sure what they mean by concurrency, I think you are right, for accessing a one resource, it should be locked anyway once pulumi up started, however at this screenshot you can see that they mention like 
For instance, you will need to manually configure secure access, encryption, and history, and devise your own concurrency control and recovery capabilities.
So I understand that I need to configure secure access, encryption at transit for s3 bucket, and pulumi up histories are stored by pulumi itself. • So for recovery part, your idea is kinda okey, I need to think about this at ci/cd platform pipelines, like what kind of implementation that I can manage, like what kind of edge cases I can face with, so thanks for the idea though, I will think on it • however by 
devise your own concurrency control
, I didn't understand what type of concurrency model they mean under the hood with 
pulumi cloud
. @victorious-church-57397 Do you have any idea ?
v
I’m not entirely sure what that means in the context of state as this is locked by running 
pulumi up
, but then again I’ve never used pulumi cloud
Which CICD tool are you using? You could implement something to snapshot state at each run, and lock your runs with concurrency so that each run has a clean rollback option
p
I see, I think your experience with s3 is really important for me. What do you think using pulumi OSS parts in production grade environment. Like since you don't use pulumi esc etc. So I think you lose some advantages, how do you manage your secrets? how do you manage your configurations that changes between environments? • I have checked some large scale organization architecture(monorepo and codebase hierarchy) but still wondering your approach For your questions: Actually I haven't implemented a CI/CD yet, I'm on the edge of using pulumi with a new project, so I'm trying to understand where pulumi fall short, so I can handle those parts myself but after it I will directly check CI/CD part for Pulumi OSS
And actually I wonder your idea here @victorious-church-57397 For example let's say if your resources has been manually changed or something happened. I think at those type of scenarios, you need to run 
pulumi refresh
and then you need to run pulumi up, because pulumi doesn't understand this. Are there any other scenarios like this, and how do you manage those type of scenarios in your CI/CD ? Do you have a working, advanced work-flow ?
v
What do you think using pulumi OSS parts in production grade environment. Like since you don't use pulumi esc etc. So I think you lose some advantages,
how do you manage your secrets?
i used to manage secrets by either: • storing them in github actions secrets and passing them to the pulumi program using environment variables • storing them in the pulumi stack as secrets • had an application secrets stack which would create entries in secretsmanager
I'm trying to understand where pulumi fall short, so I can handle those parts myself but after it I will directly check CI/CD part for Pulumi OSS
in my opinion, pulumi doesn't "fall short" in comparison to any other open source IAC tooling. what specifically are you concerned about? at my last company, we used OSS pulumi for everything from sandbox through to production and the experience was great, and we never had any issues which were directly related to using pulumi
p
Actually I'm trying to understand that since I don't have any prod. grade experience with pulumi, I wanted to see which type of problems I can face with. btw, do you use 
AWS Classic
or 
AWS Native
api ?
v
For example let's say if your resources has been manually changed or something happened. I think at those type of scenarios, you need to run 
pulumi refresh
and then you need to run pulumi up, because pulumi doesn't understand this. Are there any other scenarios like this, and how do you manage those type of scenarios in your CI/CD ?
you shouldn't be manually updating your resources in a production environment. you could mitigate behaviours like this by giving engineers ReadOnly permissions in production environments, and have a BreakGlass role you can use if disaster strikes and you have to manually edit any resources. for sandbox/dev environments where things are changed manually frequently, you should just be able to run 
pulumi refresh
/ 
pulumi up
from your local machine to fix up resources
we used a combination of AWS Classic and AWS native
p
I see, thanks for the answers @victorious-church-57397 really appreciated, it helps
i will try to implement a custom history and rollback development for our stack with using another s3 bucket, I will basically follow the idea you shared with me
I have 2 more questions, @victorious-church-57397 • Do you just use CI/CD actions of pulumi, any additional configs perhaps ? And what is your strategy for applying pulumi up, like do you trigger actions before merge(when PR opened) or after merge(when PR merged), and how do you mitigate the side effects(like if you use before merge strategy so you will end up with having different state then main branch so someone can have problem with state if they open a branch before you close merged it and for the after merge apply if your apply fails, how do you rollback, because you have a git state in main branch which is broken. • How do you apply versioning and release management? for example, at terraform, I was versioning my modules, and keeping their change logs and orchestrating to environments with these versions which was really important. So this is my question, like if you version, which part of your stack do you version, do you write custom resources and version only those and use stacks just for orchestrating purpose, or maybe something another strategy ? And how do you promote versions between different environments. Let's say you bring a new change to your eks resource and versioned it from 1.1.2 to 1.2.0 and applied it at dev environment, how do you promote to up to date version at other environments ?
v
i will try to implement a custom history and rollback development for our stack with using another s3 bucket, I will basically follow the idea you shared with me
honestly, i think that might be overkill - as long as you have a couple of environments for testing you should be fine, we never required a rollback mechanism. it would be cool to build though 🙂
i'll respond to the 2 more questions later on - just in a session
p
Thanks, but It's kinda required for my use case 🙂 I work as a consultant and I've different type of customers from medium to large scale, some of them have 7-8 domains and each domain has 3-4 environment from nonprod to prod environments, so I worked with terraform for like 3 years and I've worked on different topics and so now I'm working a self-taught pulumi workshop about the topics I want to cover on. To be honest, some of them can afford pulumi cloud but some can't, and since I'm kind of trying to be the person who wears trusted-hat for them and guide their process, It's not quite overkill for me, I follow some of the other SaaS products like env0, spacelift etc and for mature path at IaC it's kinda required for me. for example the topics(titles) I mentioned were required within my workflows while I'm working with terraform If you want to check the titles I want to focus at pulumi as well -> https://github.com/ugurcancaykara/pulumi-workshop Now I'm working on testing and gitops workflow part with pulumi, I'm trying to put an outline for managing pulumi at large scale with best practices, and that's why I asked those kind of questions 🙂
Thanks for the answers @victorious-church-57397 really appreciate it
v
thats fair enough mate - understood. let me know if you need any help with the rollback mechanism - i'd be keen to help out and get stuck in 🙂
p
I just wonder your answers about above 2 questions or any guides :)
v
will write up answers later at lunch time 🙂
p
ofc, enjoy your meal mate, appreciate it 😄
f
@victorious-church-57397 @plain-vegetable-53232 it's so great to see other people talking about this! Maybe we need a 
pulumigrunt
tool, the same way terraform had 
terragrunt
lol. In the early days before terraform supported dynamodb locking, terragrunt added that. Bookmarking state would be simillar. I also really want to do a personal hackathon where I create a full-stack app that can visualize your pulumi stacks (similar to pulumi cloud) and give you convenience URLs to the various resources. My thinking is you could run it locally to visualize your local state, and make it deployable in your GCP/AWS/Azure account as a serverless function that can read your state bucket and give you the same visuals there. I struggle to leave AWS CDK because CloudFormation gives you a "stack catalog" for free. So this would approximate that. Either of you down to try and hack a POC together? I was thinking React/TS for the frontend and Python for the backend--but could also do the whole thing in TS.
v
@plain-vegetable-53232
• Do you just use CI/CD actions of pulumi, any additional configs perhaps ? And what is your strategy for applying pulumi up, like do you trigger actions before merge(when PR opened) or after merge(when PR merged), and how do you mitigate the side effects(like if you use before merge strategy so you will end up with having different state then main branch so someone can have problem with state if they open a branch before you close merged it and for the after merge apply if your apply fails, how do you rollback, because you have a git state in main branch which is broken.
i used github actions, but not the pulumi action. i just wrote a common action which all of our stacks could use (we followed a specific naming convention for stacks of 
{env}-{repo-name}
. i had it configured so that when a PR was opened, one of the checks would be a 
pulumi preview
and you couldn't merge changes if that failed. after it was merged in, one of the steps would be to run 
pulumi up
to deploy our infrastructure/application changes. nothing should be making changes to your production environment besides CI/CD tooling. the only times we ever saw failures was in lower environments (sandbox, dev) where people had access to change things manually/from their laptops using pulumi. if we did have failures, it would be on the engineer who'd made the changes to fix up the dev environments. the only kind of failures we'd ever see in prod is if we were adding a new dependency on an output from one stack to another, and the dependent stack was attempted to be updated first.
• How do you apply versioning and release management? for example, at terraform, I was versioning my modules, and keeping their change logs and orchestrating to environments with these versions which was really important. So this is my question, like if you version, which part of your stack do you version, do you write custom resources and version only those and use stacks just for orchestrating purpose, or maybe something another strategy ? And how do you promote versions between different environments. Let's say you bring a new change to your eks resource and versioned it from 1.1.2 to 1.2.0 and applied it at dev environment, how do you promote to up to date version at other environments ?
you can write pulumi components and publish them as libraries. i had numerous internal libraries published to github actions which we could then use across our estate to ensure consistency for configurations. we didn't version stacks, i'm not entirely sure this is possible out of the box and would likely require writing a specific tool to do this. the way i managed promoting versions was to have the environment deployments chained in a github action, where the later environment depends on the previous one. everything was deployed via CI/CD except from when developing against sandbox/dev
hope this answers your questions and helps out a bit. please let me know if you have any more questions
@fierce-xylophone-92490 sure, that sounds interesting. one thing i always intended to build was an inter-stack dependency graph. inspecting the state bucket and capturing all the dependencies between stacks and visualising, then from that could drive the deployment order of stacks when provisioning a new environment/if there's a big release going out. maybe we could work that in there too?
p
@victorious-church-57397 Actually it helps 🙂 I have more questions 😄
hahah
@victorious-church-57397 Thanks for the answers, again... 1- I think the naming convention part is also crucial, so I will add into account. I understand that you guys prefer to use 
after-merge-apply
, and if it fails in the main branch, then the developer who made the changes will fix it, I think using 
git revert
manually can help (actually I was thinking rollback mechanism for such cases, btw). So basically, If I understood it correctly, you check which path has been changed, and depending on its name, you get the environment and repository information in a way that your workflow can parse and use to run in corresponding environment. Good idea! 2- I was thinking using pulumi components as well, actually I have a plan to do that and I want to ask you a couple of questions regarding on this • How do you package your components (in a different repository such as modules repository like in terraform?) ◦ For the versioning part, actually component resource is the exact codebase I want to version, I think stacks are just for you know, orchestrating and they just provide environment specific information, and I think this is where I'm gonna use versions across different environments( let's say prod has 1.0.2 of eks component but in dev stack it's 1.0.4 etc) • What kind of hierarchy do you use for this ? ◦ Monorepo or Multi repo ▪︎ I'm planning to use monorepo for convention and best productivity and actually really wonder about your thoughts here, I'm planning to store my public api(components) inside pkg folder like in this example structure -> https://github.com/ugurcancaykara/pulumi-workshop/tree/main/markdowns/large-scale-organization/centralized-example-3-for-go#an-example[…]t-structure ▪︎ There are two problem with this structure: • How to version each component differently inside pkg folder in a monorepo structure(I was using release-please for monorepo path-based versioning at my terraform modules, it can work here as well) • I'm not sure yet about how to maintain components for micro-stacks architecture -> this pattern -> https://github.com/ugurcancaykara/pulumi-workshop/tree/main/markdowns/large-scale-organization/centralized-example-2#micro-stacks ◦ For example, my core devops team can maintain the core services such as networking etc. but I may provide to developers another stack(which has more change ratio etc) and since they feel comfortable with Typescript, they may use this kind of structure, so I'm planning to work on this structure and wonder your thoughts about how to maintain multi-lang components together(it ok for just golang stack).
@fierce-xylophone-92490 I'm interested in such project as well 🙂 It would be really beneficial for open-source pulumi. So I'd be a part of such project
@fierce-xylophone-92490 looks like a good starting point, it seems exciting 🙂 I will check the video and share my notes
v
awesome news mate - i'll take a look at the x-stack deps either today or tomorrow 👍
f
It's got a sidebar now. And it's actually cloud agnostic. 
AWS_PROFILE=xxx pulumi-ui up --state-uri <s3://bucket/>
(or point at 
~/local/path/to/.pulumi-state
) now lets it actually browse the files.
v
thats the intra-stack dependency graph isn't it?
f
Yeah! Just intra-
v
looking good mate. i'm busy at work at the moment but i'll try have a look tonight
f
Now we need x-stack
v
if not will be tomorrow
26 Views
Open in Slack
PreviousNext
Powered by