Pulumi Stacks … Defined other than via CLI? We’r...
# getting-started
g
Pulumi Stacks … Defined other than via CLI? We’re just beginning to look at Pulumi - we have a long history of using CloudFormation/CDK, but we have a few projects where Pulumi would make more sense. All of our deployments are done via code, and never by humans … so for a new CloudFormation stack for example, a
cdk.Stack
is defined and added to a pipeline, which in turn automatically creates the github actions we need to create/update/delete that stack. In Pulumi, all the docs I am finding talk about initializing new stacks with
pulumi stack init
.. which seems to primarily write stack state into a state-backend (
local
, or more preferably
s3:/…
). This seems fine for initial local development and experimentation, but I feel like I’m missing some piece of the puzzle in terms of doing this programmatically for large environments. 1. While I understand that state is stored in a state backend (let’s go with S3 for now), can the definition of a new Stack be done via code? 2. Let’s say we separate Staging from Production, and we allow developers to run
pulumi stack init
for _new staging stacks_… but we want a CI/CD system to be the only thing deploying new production stacks. How is that done? 3. Is there some way to populate the list of stacks that the
pulumi stack ls
command sees via code/configuration as opposed to a remotely stored state file?
l
You're looking for the automation-api.
m
g
So I looked a bit … and I still want to maintain the idea that we use a CLI to deploy the stacks (I want to maintain the local developer feel, and also use the github pulumi action for deploys).. Is it possilbe to use the automation API just purely to ensure that the
Pulumi.$stackName.yaml
files are defined in a particular way?
l
You can use the automation-api to work with non-inline projects, which might do what you need. You could put all the default values in Pulumi.yaml, and if your logic creates a new stack, you can handle that however you want: create a new stack yaml file, don't create the stack yaml file and use the project defaults, or have sensible defaults defined in your automation-api program, and create the new stack using that.
This is a hybrid approach, where people could work with non-inline projects by bypassing the automation-api program. If you put logic in your program that you require before running operations on a stack, then you'll have to have business processes in place to handle that. Hybrid approaches allow the best of both worlds, but they also provide all the guns and feet of both worlds...
g
k thanks… i found a little bit of magic around calling he
saveStackSettings
function to generate the yaml file…
l
Cool, that's new to me
g
So for our CDK projects, we make heavy use of Projen to manage the repos … we never ever manually create .yaml/json/etc config files, it’s all done through the Projen process and a series of custom “project” types for our env. So we get all kinds of automation around dependency updates, stale branch removal processes, etc. We also get automated deployment pipeline bulding … all of the .github/workflows/* files are completely managed as part of the projen project. I’m trying to do the same thing with Pulumi… so I have a workflow now that understands how to build out the Pulumi Actions into Preview and Deploy pipelines, and for every stack (a string for now) that we pass in, it creates a new Job in each workflow. I’m just trying to get it all nailed down so that there is no human management of this stuff.
I think it might actually make more sense for me to just built the
Pulumi.XX.yaml
files directly with Projen.. Is there a schema doc for that file?
m
Oh, matt, I'm doing a very similar thing to you.
Our solution was to host a private NPM repo with a series of hosted NPX actions to configure Pulumi.**.yaml with: esc-env, region, etc.
We use GHA so here is a skeleton of what we do, in CI
Copy code
- name: Prepare Pulumi
        run: |
          cd ./.cicd/pulumi
          npm ci
          npx --yes @REACTED/REDACTED-cicd pulumi:prepareGhaJobMatrixStep
        env:
          CICD_PULUMI_STACK_NAME: ${{ matrix.tenantConfig.stack-name }}
      - name: Pulumi Update
        uses: pulumi/actions@v4
        with:
          work-dir: ./.cicd/pulumi
          command: up
          stack-name: ${{ matrix.tenantConfig.stack-name }}
Copy code
the basic prep skeleton (we use our own CLI injection drivers but this will give you a good picture:
export class PrepareGhaJobMatrixStep extends ShellCommand {

	public name = 'prepareGhaJobMatrixStep';
	public help = 'Prepares a GHA deployment step by consuming the tenant or environment partitioned output from "pulumi:createGhaJobMatrix".';
	public arguments = [{ help: 'Inputs are consumed from .env.CICD_PULUMI_STACK_NAME; this must be a fully qualified stack name. Ex.: REDACTED/pmb-main/stag. conditional: CICD_PULUMI_SHARED_SERVICES=true', synopsis: 'roleName' }];

	async action(_: CommandParameters): Promise<void> {

		const SHARED_SERVICES = !!process.env.CICD_PULUMI_SHARED_SERVICES;
		const STACK_NAME = process.env.CICD_PULUMI_STACK_NAME || '';
		console.log(`Preparing matrix deployment step for ${STACK_NAME}`);

		const PROJECT_NAME = STACK_NAME.match(/([^/]+)\/[^/]+$/)![1];
		const TENANT_OR_ENV_NAME = STACK_NAME.match(/([^/]+)$/)![1];
		const outputString = `pulumi stack output active${SHARED_SERVICES ? 'Shared' : 'Tenant'}Info --stack REDACTED/glob-infra/engineering`;
		console.log(`Executing: ${outputString}`);
		const output = execSync(outputString).toString();
		console.log(`Reesult: ***`);
		const json = JSON.parse(output);
		
		const tenantOrEnvData = json[TENANT_OR_ENV_NAME];

		if(!tenantOrEnvData.stacks[PROJECT_NAME])
			throw new Error(`Project ${PROJECT_NAME} appears to not be configured for ${TENANT_OR_ENV_NAME}`)
		
		const overridesString = (() => {
			const overrides = tenantOrEnvData.stacks[PROJECT_NAME]['overrides'];
		
			if(!overrides)
			return '';
		
			const overrideKeys = Object.keys(overrides);
			let output = `${PROJECT_NAME}:${overrideKeys[0]}: ${overrides[overrideKeys[0]]}`
			if(overrideKeys.length > 1)
				overrideKeys.splice(1).forEach(key => { output += `\n  ${PROJECT_NAME}:${key}: ${overrides[key]}` })
			return output;
		})();
		
			const escEnv = tenantOrEnvData.stacks[PROJECT_NAME]['esc-env'];
		
const escEnvString = escEnv ? `environment:
  - ${escEnv}` : ''
		
const configString = `${escEnvString}
config:
  aws:region: ${tenantOrEnvData.region}
  ${PROJECT_NAME}:mode: ${tenantOrEnvData.stacks[PROJECT_NAME].mode || 'development'}
  ${overridesString}
`;
		
		const configFileName = `Pulumi.${TENANT_OR_ENV_NAME}.yaml`
		const ecrLoginString = `aws ecr get-login-password --region ${tenantOrEnvData.region} | docker login --username AWS --password-stdin ${tenantOrEnvData.awsAccountId}.dkr.ecr.${tenantOrEnvData.region}.amazonaws.com`;
		console.log(`Executing: ${ecrLoginString}`);
		const ecrLoginResult = execSync(ecrLoginString).toString();
		console.log(`Result: ${ecrLoginResult}`);
		console.log(`Writing to file: ${configFileName}`);
		fs.writeFileSync(configFileName, configString, 'utf-8');
		console.log(`Setup Complete!`);
	}

}
^ This command sources dynamic CICD and Project ENV configuration from Pulumi ESC and writes it into a Pulumi.**.yaml file for the correct stack being deployed.
g
interesting… you should look at Projen… i think you might like it
m
Will have to check it out. Hopefully I've been of help!
g
https://github.com/nextdoor/cdk-pipelines-github < we forked the mostly dead cdk library here into our own, and we use this to manage a dozen CDK repos pipelines..
m
Our pattern is to execSync pulumi esc env command configs and build files based on those
g
but this, in concept, is what i want for pulumi too
m
our entire CICD pipe is actually dynamic based on ESC configs and envs
so we never change any CICD code - just tweak ESC
this is for a multi-tenant, multi-env (3d matrix) setup
we have a trigger service that polls ESC and kicks off builds as necessary (which is a bit hacky but it works quite well for us) - especially because we use a pattern of, in our stacks: • ingest ESC config. process. create infra. stack output hydrated data. • next stack consumes the previous output, etc. so our little hack-a-crap service triggers the correct CICD build chains based on ESC updates
a bit ghetto and its' up for immediate chopping block but it has served its purpose quite well
our pulumi process uses a lot of very heavy stuff to deploy and update ML models so we have to have them on custom 128cpu / 512GB ram runners
pulumi deployments were not an option
g
one othe rquick question while i have you… if you have a repo with multiple stacks, how do you control the “entrypoint” for a stack… like which path does it go down to start defining resources? i figured there’s be some
entrypoint
key in the
Pulumi.$stackName.yaml
file.. i dont see that.
m
oh yeah, that's in our preparematrixjob NPX
Copy code
export class CreateGhaJobMatrixCommand extends ShellCommand {

	public name = 'createGhaJobMatrix';
	public help = 'Prepares a GHA deployment matrix. Downstream consumer: "pulumi:prepareGhaJobMatrixStep" ESC namespace: "glob-infra:tenants" OR "glob-infra:shared';
	public arguments = [{ help: 'Inputs are consumed from env.CICD_PULUMI_INCLUDE_TENANTS/ENVIRONMENTS, env.CICD_PULUMI_EXCLUDE_TENANTS/ENVIRONMENTS and env.CICD_PULUMI_PROJECT_NAME conditional: CICD_PULUMI_SHARED_SERVICES=true', synopsis: 'roleName' }];

	async action(_: CommandParameters): Promise<void> {

		const SHARED_SERVICES = !!process.env.CICD_PULUMI_SHARED_SERVICES;
		const whitelist = SHARED_SERVICES ? process.env.CICD_PULUMI_INCLUDE_ENVIRONMENTS : process.env.CICD_PULUMI_INCLUDE_TENANTS; 
		const blacklist = SHARED_SERVICES ? process.env.CICD_PULUMI_EXCLUDE_ENVIRONMENTS : process.env.CICD_PULUMI_EXCLUDE_TENANTS;
		const PROJECT_NAME = process.env.CICD_PULUMI_PROJECT_NAME || ''; 

		if (!PROJECT_NAME || PROJECT_NAME === '') {
			throw new Error('CICD_PULUMI_PROJECT_NAME environment variable required');
		}

		const output = execSync(`pulumi stack output active${SHARED_SERVICES ? 'Shared' : 'Tenant'}Info --stack REDACTED/glob-infra/engineering`).toString();

		const json = JSON.parse(output);

		const activeTenantsOrEnvironments = Object.keys(json);

		const applicableTenants = (() => {
			if (whitelist && blacklist) {
				throw new Error('Please use only one of: CICD_PULUMI_INCLUDE_TENANTS, CICD_PULUMI_EXCLUDE_TENANTS');
			}

			if (whitelist) {
				return activeTenantsOrEnvironments.filter(t => whitelist.includes(t));
			} else if (blacklist) {
				return activeTenantsOrEnvironments.filter(t => !blacklist.includes(t));
			}
			return activeTenantsOrEnvironments;
		})();

		const pulumiStackList = execSync('pulumi stack ls').toString();

		const extractStacks = (output: string): string[] => {
			return output.split('\n')
				.filter(line => line.includes(PROJECT_NAME))
				.map(line => {
					const parts = line.split('/');
					return parts[parts.length - 1].trim();
				});
		};

		const existingEnvironmentsOrTenants = extractStacks(pulumiStackList);

		const matrixJobArrayJson: {
			'aws-region': string,
			'aws-role': string,
			'stack-name': string,
			'tenant'?: string,
			'environment'?: string,
		}[] = [];

		applicableTenants.forEach(tenantOrEnvironment => {
			const tenantData = json[tenantOrEnvironment];

			if (!tenantData.stacks[PROJECT_NAME]) {
				return;
			}

			if (!existingEnvironmentsOrTenants.includes(tenantOrEnvironment)) {
				execSync(`pulumi stack init REDACTED/${PROJECT_NAME}/${tenantOrEnvironment}`);
			}

			matrixJobArrayJson.push(SHARED_SERVICES ? {
				'aws-region': tenantData.region,
				'aws-role': tenantData.iamRoleArn,
				'stack-name': `REDACTED/${PROJECT_NAME}/${tenantOrEnvironment}`,
				'environment': tenantOrEnvironment.toUpperCase(),
			} : {
			
				'aws-region': tenantData.region,
				'aws-role': tenantData.iamRoleArn,
				'stack-name': `REDACTED/${PROJECT_NAME}/${tenantOrEnvironment}`,				
				'tenant': tenantOrEnvironment.toUpperCase(),
			});
		});
		console.dir(matrixJobArrayJson, { depth: null });
	}

}
this also creates/destroys the appropriate stacks
^ the output of that is consumed by the step i linked earlier to configure/set the right stack and create the Pulumi.STACK_NAME.yaml for it
g
interesting… it seems to me like the pulumi team doesnt have a strongly opinionted way you do this, so they kind of leave it up to you..
l
@great-sundown-78827 Stacks don't have entrypoints, projects do. They use index.ts,
__main__.py
, or whatever your preferred language's default entrypoint.
m
^ would trust him over me 😄
l
I think I was answering a different question 🙂
m
Side note - it would be very interesting to have (if it doesnt exist already) a bridge between Pulumi stack outputs and cloudformation stack outputs (outside of those set inside the stack)
l
Do you have an example? It may already exist!
m
Nope 😂 I know we can manually do it through pulumi via custom resources but i havent had the time to write it
l
I mean: explain what you mean, use cases, or something like that 🙂
m
Oh, yeah. • at start of stack, we initialize: ◦ const sharedConfig = new PulumiCloudformationConfigBridge(blah blah) • during the build, it contains a hybrid of specified stack outputs and cloudformation outputs • any edits are persisted. TLDR sync
would be very handy for integrating with non-pulumi cloudformation things
l
Ah. You want inputs from multiple sources, like stack references for both Pulumi and CF.
m
Yeah, I know we can technically do so inside ESC by using CF as a provider
but that method is extremely painful
and it's also one-way.
l
Yea, that was what I was going to mention. If it's painful, then let the Pulumi team know!
m
we've wanted to use pulumi in conjunction with SST/serverless for example, quite a few times.
ESC is still in preview, those guys are pretty smart; im sure someone's already complained 😂
From what i understand the entire provider architecture of ESC was never meant for two-way sync (ESC itself wasnt meant for it)
Which is a design choice that 100% makes sense for more reasons than i can count
l
Two-way sync sounds like a design concern. Single source of truth, acyclic graphs and all that? If two systems need information from each other, then (imo) you need to split at least one of those systems in two.
m
yup!
it already can be acyclic if your ESC uses CF as a provider and your stack writes to CFN outputs 🫠
personally keeping my orgs out of that hot potato haha
l
That sounds cyclic to me 🙂
m
whoops. that one. circular dependency as we'd call it in our circles, but w data
g
@little-cartoon-10569 Since you’re here… I think I’ll take a second stab at my questions as someone coming from a CDK/CloudFormation world. Maybe you can help connect a few dots I’m missing. CloudFormation… In CloudFormation you deal with “Stacks” and “Templates”. Templates define what a stack might look like, along with the combination of parameters supplied to that instance of the stack which control behavior that was defined by the templates. We have 10+ years running many thousands of cloudformation stacks all defined in plain YAML. Over the last few years we’ve been moving to CDK … and CDK really just programmatically generates the “Templates” on-demand … and you sort of lose track of that concept and really just focus on writing “Stacks”. I can define a stack like this:
Copy code
const stack = new Stack(app, 'MyStackName')
new Bucket(stack, 'MyBucket')
And that’s really it … you then run
cdk ls
and you see
MyStackName
.. and
cdk deploy MyStackName
will deploy it. No state management, no pre-creation of a stack via a CLI … everything is in code. In fact, we regularly define new stacks just by editing a list or map in a typescript file in the Github UI, and creating a PR … Now Pulumi… I am trying to map this to how Stacks work in Pulumi. I get that Pulumi needs to maintain state … so sure, we populate an S3 bucket for that. I also get that to really deploy a Stack, it needs to write actual state into the bucket.. fine. For developing though.. it feels like I am missing something. I want something akin to the
new Stack(…)
concept, where we define a
Stack
as a typescript class, then add various Pulumi components into the stack to define what is managed in that stack. Is there some way of writing Pulumi code in this way, where we can define these top level contructs like a
PulumiStack
and then place objects onto that stack? all without calling the CLI to pre-create some YAML file… that frankly doesnt seem to really matter that much?
l
You don't need a YAML file for a stack or project. You can do it all in automation-api, and you don't need to write anything to YAML.
It seems like Pulumi projects match templates. But I'm not sure the CF stacks are the same as Pulumi stacks. Stacks are instances of a project, and the differences between projects is only configuration: e.g. which region they're deployed to, whether or not redundant DBs are required, that sort of thing.
In Pulumi without automation-api, the flow is just like you described, but the stack creation part might happen in a pipeline, based on a branch name. It's all in code, just not all in the same code file.
In automation-api, you can put it all in the same code file, if you like.
g
Stacks are instances of a project
… that is a significant distinction.. I think I want to touch on that for a sec. I imagined that
Stacks
are a sub comonent of a
Project
which is a component of an
Organization
. So I figured a single Git repo might have several stacks that focus on different thigns… a
PagerdutyStack
and a
GrafanaStack
for example. I hear you saying that a
Stack
might be a
Dev
version of
projectA
and a
Prod
version of
projectA
.. but fundamentally the code is the same and roughly the resources they launch are the same. Am I getting that right?
In our current development case - we are looking to manage Grafana, PagerDuty and Datadog configs… not things we’re likely to have a significant distinction on between “Dev” or “Staging” in our case. Ie. .. we’re not going to create a
FooTeam
for “staging” vs “dev” vs “prod”.. its one team. We might create multiple Services (eg
Foo (Stg)
,
Foo (Prd)
) - but we don’t really have a need or reason to have say two different deployment pipelines for that. So given that model… it sounds like we might just equate a
Project == Stack
in that case.
l
Yes, a stack might be a dev version vs. a prod version, or an EU instance vs. a US instance.
You might have projects with only 1 stack, but generally not. You'll always want at least a staging/dev instances, and a prod instance. You may destroy the dev instance as soon as you've proved your code correct, but you'll still want it.
I don't generally split projects based on service (Grafana, Datadog, etc.). I split them based on deployment cycle. If you deploy your Grafana and Datadog projects once in every blue moon, you might want to put them in the same project. And your back-office app might be getting frequent updates based on your in-house development cycle, so that would be in a different project.
g
Yeah that makes sense… A project owuld be a git repo in our case, and we’re planning on the Datadog/Grafana/PagerDuty stuff being mostly there… so I agree with that thought process. I think that distinction on the
Stack
was important for my mental model.
l
It is.
g
im looking at trying t mix the automation API and Projen together with something like this:
Copy code
/** @format */

import { Component } from 'projen';
import { InlineProgramArgs, LocalWorkspace, PulumiFn } from '@pulumi/pulumi/automation';
import { PulumiProject } from './pulumi-project';

export interface PulumiStackOptions {
  readonly secretsProvider: string;
  readonly program: PulumiFn;
}

export class PulumiStack extends Component {
  public readonly stackName: string;
  public readonly projectName: string;
  public organization: string = 'organization';

  protected readonly secretsProvider: string;
  protected readonly program: PulumiFn;

  constructor(project: PulumiProject, id: string, options: PulumiStackOptions) {
    super(project, id);
    this.stackName = id;
    this.projectName = project.name;
    this.secretsProvider = options.secretsProvider;
    this.program = options.program;
  }

  synthesize(): void {
    this.asyncSynth().catch((err) => {
      console.error('ERROR DURING SYNTH');
      console.log(err);
      throw err;
    });
  }

  async asyncSynth(): Promise<void> {
    const args: InlineProgramArgs = {
      stackName: this.stackName,
      projectName: this.projectName,
      program: this.program,
    };

    const stack = await LocalWorkspace.createOrSelectStack(args, {
      workDir: '.',
    });
    await stack.workspace.saveStackSettings(`${this.organization}/${this.projectName}/${this.stackName}`, {
      secretsProvider: this.secretsProvider,
    });
  }
}
so running
npx projen
which sets up the rest of the repo, would also set up the
Pulumi.xxx.yaml
contents, as well as ultimately handle creation of the github deploy pipeline
but im ultimately not sure that the automation api is what i want here… i think perhaps i just want to use Projen to manage the
Pulumi.yaml
and
Pulumi.$stackName.yaml
files…