Hi all, I'm using S3 as my backend and I specify the s3 path in

Pulumi.yaml

I want the backend s3 bucket to be different for each environment. How do I do it? Initially I wanted to use the same bucket name in each environment but then remembered that bucket names are global.

name: pulumi_backend
stackConfigDir: ./config
runtime:
  name: python
description: S3 backend for self-hosted Pulumi
backend:
  url: <s3://fd-pulumi-backend>

basically this needs to be different for each stack (

qa

,

prod

, etc.):

backend:
  url: <s3://fd-pulumi-backend>

You should not set the backend URL in the project file (

Pulumi.yaml

). Instead, simply do a

pulumi login s3://<bucket>

and then run

pulumi stack init <stack name>

. This way, each stack is initialized in a different bucket.

Did you know that you can put a path in the backend? That would allow you to use one bucket for all your state info, but keep each state separate within that bucket.

True. Unless you want to restrict access to stacks by the accounts in which the buckets are.

The backend account / creds should be separate from the stack account / creds. There's no need to have any relationship between state infra and stack infra.

I agree with you. But if you want to restrict access to your prod stack's state (simulate Pulumi Cloud's RBAC), you may want to have that in a separate bucket. I am sure there are other ways to manage access to objects in the same bucket though. For instance, you could also have an IAM policy restricting certain path prefixes I think.

Thanks! how do we define the backend in the stack file? I want to avoid running

pulumi login

every time I need access to a specific backend...

Just do it once, see what gets put into the stack file, and copy/edit accordingly? I'm not sure it even goes in the stack file, it might go in the profile in ~/.pulumi. Praneet probably can confirm.

Yeah I don't believe the backend URL goes in the stack config file at all.

when you do login, nothing goes to the stack file

If the experience of

pulumi login

is not up to your liking for switching between stacks (which can be annoying) you should do what @little-cartoon-10569 is suggesting.

not sure I'm following @little-cartoon-10569’s method... any info in the docs?

See :https://www.pulumi.com/docs/concepts/state/

The bucket-name value can include multiple folders, such as my-bucket/app/project1. This is useful when storing multiple projects’ state in the same bucket.

Ahh, I can't use the same bucket for all projects because each project is for a separate AWS account

I recommend treating state infra as if the only user who can ever access it is your CD pipeline user. This makes life much easier. It does move access control to business processes, which can concern some people, but I have never found a case in which the extra effort to duplicate app.pulumi.com's features have been worth the time invested.

ahhh I think I get what you mean....

You can put the special backend creds in the backend URL (or pass to pulumi login):

As of Pulumi CLI v3.33.1, instead of specifying the AWS Profile, add awssdk=v2 along with the region and profile to the query string. The URL should be quoted to escape the shell operator &, and used as follows:

pulumi login 's3://<bucket-name>?region=us-east-1&awssdk=v2&profile=<profile-name>'

(side note: I actually wish the backend URL was stored in the stack config. Both for self-managed as well as for Pulumi Cloud-managed stacks. For the latter it would be great if the stack config file showed the username or org under which the stack exists.)