https://pulumi.com logo
Join Slack
Powered by
What would be a recommendation for state managemen...
# getting-started
s
What would be a recommendation for state management in terms of having multiple environments (dev, test, prod). Should each environment have a separate Storage Account for each state, or have one Storage Account state for all environments? • Right now I have first, but find out that managing multiple environments stacks (pulumi.dev.yaml, pulumi.test.yaml, pulumi.prod.yaml) on a local machine starts to be problematic, as you need to log out and log in to the appropriate backend before calling operation on any stack. If you forget about it - pulumi will start operating on the last logged-in, which may be an incorrect one. • I used a different URL, in each env stack, but that seems to be ignored once you are logged in pulumi
Copy code
backend:
url: <azblob://pulumi-state-container?storage_account=poc0pulstate0swc0dev>
t
What exactly is wrong with just having separate stacks on the same account?
s
Maybe not wrong, but we have this first gut feeling if we have different Azure subscriptions for each environment, that it should also be reflected by the backend storage -> having each in an appropriate subscription. But just found that it makes things a bit more problematic. And wonder what is common/recommended practice here.
t
I'm not sure I can speak to common or recommended practice, I'm relatively new to the community myself if I'm honest. I would say those stocks are built-in feature specifically designed to handle this kind of thing, and if you need different accounts or some way of signifying that you can always add in a variable into your stack configuration that you can utilize to tag or something of that nature would that solve your use case?
s
Not really. It is about how pulumi login works. When I log in to the backend for DEV, then if I run stack TEST - it just will work. But will write to the state on the DEV subscription. I need to pulumi log out and log in to TEST subscription state.
t
Yeah but each stack has its own State doesn't it
So why not just utilize a different stack for every type of deployment you have and call it a day, why do they need to be separate accounts
s
See three reasons (not hard, just trying to challenge): • with one state backend - it must lay in some subscription. Probably had to create separate subscription for that • SPOF • possibility (stupidity, but it is possible) to reference stack from another environment 😄
t
Separate stacks are as separate as separate accounts they still run in Pulumi cloud, what's the fail case, I must not actually fully understand the situation
s
I am not using Pulumi Cloud, I am using Storage Account as state backend
t
Ohhhh
In that case I'm at a loss
Please share your findings when you're done!
f
yeah in my (biased) opinion having to 
pulumi logout
and 
pulumi login
to switch here is the biggest user pain in that approach, but tbh if that's what you and your org have decided works best for your requirements, that's how it is. note I think for 
backend: url:
stack config to work you have to be logged out but I haven't tested this. ultimately, this is all part of why orgs pay for pulumi cloud as all this is handled for you (and checkpoints, and backups, and etc) so you can get to developing your solution instead of spending time and effort managing your own backend
v
In my previous organisation (we didn’t use pulumi cloud) we had separate accounts and had bash aliases to authenticate to the relevant accounts to access the backend. So we’d type “prod” and we’d be in the prod account, or “dev” for the dev account. Sure it’s not perfect but it’s a lot better than running the authentication and login commands each time
f
same, yeah. definitely worth setting up some automation there.
v
we also had an optional argument for stackname, if the stack name was specified, it would login to the relevant account and then select the stack that was passed in. made things a lot nicer
s
we have already something like that, but that limits native API
if I want to run other pulumi command - I would need to wrap all of them
v
im not too sure what you mean, could you elaborate?
s
let say that frequent usage is pulumi preview
I can make an alias/script like mypulumi preview --stackName which will select the appropriate backend and login, and then run the pulumi preview underhood
But then if i would like to use other pulumi commands, on different stack/backend - I would need to wrap them too
v
why would you do that? wouldnt you just wrap the authentication/pulumi login to the backend and then just use the native commands?
s
I can, but if I forget to switch and run UAT stack while already logged into DEV backend - it may lead to many problems
v
why dont you amend your prompt in your shell to display which environment you're in?
s
one which I hit - everything wen smooth (beucase stack was new) - but the backend which was used was DEV
When I know this - i can remember, but when I want introduce this tool to 100 developers, I need some more idiotic-proof solution 😄
It just easy to forget or make mistake
you are working with DEV stack, you are logged in to DEV backend, then switching to UAT stack just by running any pulumi command - and you are doomed 😄
v
you can implement controls (such as we did) where the pulumi role for higher environments didn't have write access in AWS then you can't run pulumi locally, and we used to let CI handle it
you're always going to run in to issues like this when there's multiple accounts/environments and there's humans involved unfortunately
s
I know, that is why I am trying to figure out how it should be designed to lower the chance of mistake
But you had also multiple backends? Each for each environment?
v
yes
s
So, Prod backend was limited only to read?
v
we didnt really have any issues such as you're describing
the IAM role we used as individuals didn't have write access so if we tried to run pulumi up from our machines, we got permissions errors
there was a break glass role in case we ever needed to, and only CI could deploy to production
s
So, in summary, you had: • pulumi.yaml, pulumi.dev.yaml, pulumi.test.yaml, pulumi.prod.yaml • separate backend for each env ?
v
yes
although the config files matched the stack names
s
and how did you handle switching locally between backends?
v
s
so each time you had to remember to use that bash alias to switch account? (when you moving from one env stack to another?)
v
yeah, but how frequently are you switching environments in reality? we'd test our changes in sandbox from local, make sure they worked, perhaps in dev from local too if we wanted to, but the rest of the deployments were handled by CI, sounds like you might want to invest some time in to the development process and testing process as well as trying to find a solution to you running commands against the wrong envs
so in my above example theres only 1 enviornment change
s
Make sense, thanks for your input 🙂
Oh, one thing - how did you handle in such case, provisioning stacks for higher envs?
Through CI?
Let's say that you have microservice, and you want to introduce IAC in it. With separate backend, and limited access (let's say that only for DEV, you have write permissions) - you would not be able to setup stacks for UAT and XYZ on local machine
beucase pulumi stack init will fail due to lack of permissions
v
yeah, we had a "create stack" github action
which used the CI permissions, thats how we managed stack creation.
s
Thanks again for sharing this
v
No problem mate
Feel free to dm me if you have any other queries
14 Views
Open in Slack
PreviousNext
Powered by