hi, is there a way to use stack references when storing state in s3 buckets?

yup, seen that documentation already, that sadly doesn't explain in enough detail how to reference stacks in and s3 bucket.

I guess that you can use variables in your pulumi stack as references. It is not connected directly with your bucket object but it could be connected to any resource inside of your stack, maybe this post can help you: https://blog.ediri.io/quick-bites-of-pulumi-stack-references

yeah, seen that one already, but that doesn't show you how to use an s3 bucket as the state backend...

You need to set

PULUMI_BACKEND_URL

. To be the url to your bucket eg

<s3://my-backend-bucket/>

and then you can use stackReference to reference any of the stacks that use that bucket by doing something like

const networkingStack = new pulumi.StackReference('networking');
const vpcId = networkingStack.requireOutput('vpcId');

does that imply you only do a pulumi login to one bucket folder for different apps?

thats the way we have everything set up, we just use one state bucket for our environments and then reference the stacks from the root of the bucket as thats where the states stored 🙂

we need to separate apps and their respective stacks to avoid a proliveration of stacks of other apps/projects

we havent experimented with splitting out in to separate directories for different stacks so cant say how that would work for sure

thanks, we have to keep it separate, risk is to high with mixing stack states...

The state of each stack is separate in its own right so you can't corrupt other stacks state if you break something in another… it's similar to having a terraform s3 backend

When using S3 backend, stack references only work across stacks whose states are in the same S3 bucket. FWIW, the Pulumi service helps for this use-case since you can manage who can update which stacks via RBAC and stack references work across the organization.

yes, but if you accidentally use the wrong stack in the wrong app you can easily shoot yourself in the foot... All of our stacks share the same bucket but not the same directory, depending on the pulumi login url being used...

Not sure I'm following. When you say “stack names showing up across all apps” what do you mean by “apps” in this context? Stacks are managed on a project by project basis. A “pulumi up” will only update the currently selected stack.

You cannot view stacks across backends. To access a stack reference from a stack that's stored in at a particular S3 URL, the referencing stack must be in the same S3 URL.

@witty-candle-66007 app = project in our language… so an app can have multiple stacks .i.e variation or environments. @little-cartoon-10569 yes, that is what we discovered… Ideally the StackReference would support using different projects as references if they are in the same s3 bucket… we are finding it quite the hindrance currently that is really trivial in terraform…

I'm confused by the last paragraph. Stack references do support different projects (stacks) in the same bucket. Can you elaborate on the hindrances you are encountering.

we have one bucket, different projects. we login to one project using

pulumi login <s3://bucket/projectA>

then we do a

pulumi stack select staging

to ensure we chose the staging stack and continue to perform our pulumi workflow on that stack. Now we need to use projectB; we do a

pulumi login <s3://bucket/projectB>

etc. pp. Now if projectB needs to reference an output in projectA it currently is not possible…

Hi, @cold-toothbrush-60276 sorry for my long post but I guess it can help you if I share how I’m working: I have a seed project; in that project, I have my storage bucket to store all Pulumi states for all stacks and a secrets management to store all secrets necessary to run the Pulumi IaC. However, in my case, I have different solutions for different problems, for example: 1. New resources are managed by Pulumi IaC that runs in my CI/CD pipelines and my repository(I don’t depend on any SRE laptop or local environment to continue working, non-blocking process). This pipeline is used to onboard or offboard new services or applications. Developer A needs a new K8s cluster, just send a pull request to my repository with a YAML file that describes your desired resource, and my pipelines will provide any environment or solution and verify if the request respects internal security and governance. Self-service Infrastructure. Using this repository, pipeline, and Pulumi IaC is not frequent because most developers don’t need a new environment; they work in their environments. 2. Conciliation, to promote helm charts(releasing process) between development to production environments, developers only use their repositories and environments. Each team repositories have some branches that could be conciliated with respective environments; for example, Repository “A” has a “dev” branch that could be conciliated automatically with the “dev” environment using Pulumi Kubernetes Operator. In that use, developers run their pipelines to change these branches, and the GitOps strategy works smoothly, syncing a branch with an environment. SREs can use Python, Golang, etc... in the SRE repository described in item 1, and Developers(item 2) is using Yaml to express their needs, and Pulumi Kubernetes Operator will attend them.

@mammoth-electrician-64525 thanks for sharing. we’re not quite “there” yet. we are working in a completely isolated environment, sandboxed, doing RnD. Our use case is that one pulumi project needs to be able to do a lookup on another adjacent project to get some required info without us having to hard code these configurations into any stack specific pulumi.yaml. we’ve now resorted to just using the aws api to do a lookup and reconstruct the payload we need.

It doesn't support this. The login URL has to be the same for all projects if they're to use stack references. It's not just the bucket that has to match, it's the entire URL.

yup, that is what we discovered. and that is why we are not going to use this “feature”, we don’t want the stacks “mingling” across projects.

You can adopt the Pulumi service, and you'll get the feature back!