great-sundown-78827
02/18/2024, 5:58 PMcdk.Stack
is defined and added to a pipeline, which in turn automatically creates the github actions we need to create/update/delete that stack.
In Pulumi, all the docs I am finding talk about initializing new stacks with pulumi stack init
.. which seems to primarily write stack state into a state-backend (local
, or more preferably s3:/…
). This seems fine for initial local development and experimentation, but I feel like I’m missing some piece of the puzzle in terms of doing this programmatically for large environments.
1. While I understand that state is stored in a state backend (let’s go with S3 for now), can the definition of a new Stack be done via code?
2. Let’s say we separate Staging from Production, and we allow developers to run pulumi stack init
for _new staging stacks_… but we want a CI/CD system to be the only thing deploying new production stacks. How is that done?
3. Is there some way to populate the list of stacks that the pulumi stack ls
command sees via code/configuration as opposed to a remotely stored state file?little-cartoon-10569
02/18/2024, 6:19 PMmagnificent-soccer-44287
02/18/2024, 11:07 PMgreat-sundown-78827
02/19/2024, 8:08 PMPulumi.$stackName.yaml
files are defined in a particular way?little-cartoon-10569
02/19/2024, 8:32 PMlittle-cartoon-10569
02/19/2024, 8:34 PMgreat-sundown-78827
02/19/2024, 8:43 PMsaveStackSettings
function to generate the yaml file…little-cartoon-10569
02/19/2024, 8:44 PMgreat-sundown-78827
02/19/2024, 8:45 PMgreat-sundown-78827
02/19/2024, 8:50 PMPulumi.XX.yaml
files directly with Projen.. Is there a schema doc for that file?great-sundown-78827
02/19/2024, 8:58 PMmagnificent-soccer-44287
02/19/2024, 8:59 PMmagnificent-soccer-44287
02/19/2024, 8:59 PMmagnificent-soccer-44287
02/19/2024, 9:00 PM- name: Prepare Pulumi
run: |
cd ./.cicd/pulumi
npm ci
npx --yes @REACTED/REDACTED-cicd pulumi:prepareGhaJobMatrixStep
env:
CICD_PULUMI_STACK_NAME: ${{ matrix.tenantConfig.stack-name }}
- name: Pulumi Update
uses: pulumi/actions@v4
with:
work-dir: ./.cicd/pulumi
command: up
stack-name: ${{ matrix.tenantConfig.stack-name }}
magnificent-soccer-44287
02/19/2024, 9:01 PMthe basic prep skeleton (we use our own CLI injection drivers but this will give you a good picture:
export class PrepareGhaJobMatrixStep extends ShellCommand {
public name = 'prepareGhaJobMatrixStep';
public help = 'Prepares a GHA deployment step by consuming the tenant or environment partitioned output from "pulumi:createGhaJobMatrix".';
public arguments = [{ help: 'Inputs are consumed from .env.CICD_PULUMI_STACK_NAME; this must be a fully qualified stack name. Ex.: REDACTED/pmb-main/stag. conditional: CICD_PULUMI_SHARED_SERVICES=true', synopsis: 'roleName' }];
async action(_: CommandParameters): Promise<void> {
const SHARED_SERVICES = !!process.env.CICD_PULUMI_SHARED_SERVICES;
const STACK_NAME = process.env.CICD_PULUMI_STACK_NAME || '';
console.log(`Preparing matrix deployment step for ${STACK_NAME}`);
const PROJECT_NAME = STACK_NAME.match(/([^/]+)\/[^/]+$/)![1];
const TENANT_OR_ENV_NAME = STACK_NAME.match(/([^/]+)$/)![1];
const outputString = `pulumi stack output active${SHARED_SERVICES ? 'Shared' : 'Tenant'}Info --stack REDACTED/glob-infra/engineering`;
console.log(`Executing: ${outputString}`);
const output = execSync(outputString).toString();
console.log(`Reesult: ***`);
const json = JSON.parse(output);
const tenantOrEnvData = json[TENANT_OR_ENV_NAME];
if(!tenantOrEnvData.stacks[PROJECT_NAME])
throw new Error(`Project ${PROJECT_NAME} appears to not be configured for ${TENANT_OR_ENV_NAME}`)
const overridesString = (() => {
const overrides = tenantOrEnvData.stacks[PROJECT_NAME]['overrides'];
if(!overrides)
return '';
const overrideKeys = Object.keys(overrides);
let output = `${PROJECT_NAME}:${overrideKeys[0]}: ${overrides[overrideKeys[0]]}`
if(overrideKeys.length > 1)
overrideKeys.splice(1).forEach(key => { output += `\n ${PROJECT_NAME}:${key}: ${overrides[key]}` })
return output;
})();
const escEnv = tenantOrEnvData.stacks[PROJECT_NAME]['esc-env'];
const escEnvString = escEnv ? `environment:
- ${escEnv}` : ''
const configString = `${escEnvString}
config:
aws:region: ${tenantOrEnvData.region}
${PROJECT_NAME}:mode: ${tenantOrEnvData.stacks[PROJECT_NAME].mode || 'development'}
${overridesString}
`;
const configFileName = `Pulumi.${TENANT_OR_ENV_NAME}.yaml`
const ecrLoginString = `aws ecr get-login-password --region ${tenantOrEnvData.region} | docker login --username AWS --password-stdin ${tenantOrEnvData.awsAccountId}.dkr.ecr.${tenantOrEnvData.region}.amazonaws.com`;
console.log(`Executing: ${ecrLoginString}`);
const ecrLoginResult = execSync(ecrLoginString).toString();
console.log(`Result: ${ecrLoginResult}`);
console.log(`Writing to file: ${configFileName}`);
fs.writeFileSync(configFileName, configString, 'utf-8');
console.log(`Setup Complete!`);
}
}
magnificent-soccer-44287
02/19/2024, 9:02 PMgreat-sundown-78827
02/19/2024, 9:02 PMmagnificent-soccer-44287
02/19/2024, 9:02 PMgreat-sundown-78827
02/19/2024, 9:03 PMmagnificent-soccer-44287
02/19/2024, 9:03 PMgreat-sundown-78827
02/19/2024, 9:03 PMmagnificent-soccer-44287
02/19/2024, 9:03 PMmagnificent-soccer-44287
02/19/2024, 9:03 PMmagnificent-soccer-44287
02/19/2024, 9:04 PMmagnificent-soccer-44287
02/19/2024, 9:05 PMmagnificent-soccer-44287
02/19/2024, 9:06 PMmagnificent-soccer-44287
02/19/2024, 9:06 PMmagnificent-soccer-44287
02/19/2024, 9:06 PMgreat-sundown-78827
02/19/2024, 9:07 PMentrypoint
key in the Pulumi.$stackName.yaml
file.. i dont see that.magnificent-soccer-44287
02/19/2024, 9:07 PMmagnificent-soccer-44287
02/19/2024, 9:08 PMexport class CreateGhaJobMatrixCommand extends ShellCommand {
public name = 'createGhaJobMatrix';
public help = 'Prepares a GHA deployment matrix. Downstream consumer: "pulumi:prepareGhaJobMatrixStep" ESC namespace: "glob-infra:tenants" OR "glob-infra:shared';
public arguments = [{ help: 'Inputs are consumed from env.CICD_PULUMI_INCLUDE_TENANTS/ENVIRONMENTS, env.CICD_PULUMI_EXCLUDE_TENANTS/ENVIRONMENTS and env.CICD_PULUMI_PROJECT_NAME conditional: CICD_PULUMI_SHARED_SERVICES=true', synopsis: 'roleName' }];
async action(_: CommandParameters): Promise<void> {
const SHARED_SERVICES = !!process.env.CICD_PULUMI_SHARED_SERVICES;
const whitelist = SHARED_SERVICES ? process.env.CICD_PULUMI_INCLUDE_ENVIRONMENTS : process.env.CICD_PULUMI_INCLUDE_TENANTS;
const blacklist = SHARED_SERVICES ? process.env.CICD_PULUMI_EXCLUDE_ENVIRONMENTS : process.env.CICD_PULUMI_EXCLUDE_TENANTS;
const PROJECT_NAME = process.env.CICD_PULUMI_PROJECT_NAME || '';
if (!PROJECT_NAME || PROJECT_NAME === '') {
throw new Error('CICD_PULUMI_PROJECT_NAME environment variable required');
}
const output = execSync(`pulumi stack output active${SHARED_SERVICES ? 'Shared' : 'Tenant'}Info --stack REDACTED/glob-infra/engineering`).toString();
const json = JSON.parse(output);
const activeTenantsOrEnvironments = Object.keys(json);
const applicableTenants = (() => {
if (whitelist && blacklist) {
throw new Error('Please use only one of: CICD_PULUMI_INCLUDE_TENANTS, CICD_PULUMI_EXCLUDE_TENANTS');
}
if (whitelist) {
return activeTenantsOrEnvironments.filter(t => whitelist.includes(t));
} else if (blacklist) {
return activeTenantsOrEnvironments.filter(t => !blacklist.includes(t));
}
return activeTenantsOrEnvironments;
})();
const pulumiStackList = execSync('pulumi stack ls').toString();
const extractStacks = (output: string): string[] => {
return output.split('\n')
.filter(line => line.includes(PROJECT_NAME))
.map(line => {
const parts = line.split('/');
return parts[parts.length - 1].trim();
});
};
const existingEnvironmentsOrTenants = extractStacks(pulumiStackList);
const matrixJobArrayJson: {
'aws-region': string,
'aws-role': string,
'stack-name': string,
'tenant'?: string,
'environment'?: string,
}[] = [];
applicableTenants.forEach(tenantOrEnvironment => {
const tenantData = json[tenantOrEnvironment];
if (!tenantData.stacks[PROJECT_NAME]) {
return;
}
if (!existingEnvironmentsOrTenants.includes(tenantOrEnvironment)) {
execSync(`pulumi stack init REDACTED/${PROJECT_NAME}/${tenantOrEnvironment}`);
}
matrixJobArrayJson.push(SHARED_SERVICES ? {
'aws-region': tenantData.region,
'aws-role': tenantData.iamRoleArn,
'stack-name': `REDACTED/${PROJECT_NAME}/${tenantOrEnvironment}`,
'environment': tenantOrEnvironment.toUpperCase(),
} : {
'aws-region': tenantData.region,
'aws-role': tenantData.iamRoleArn,
'stack-name': `REDACTED/${PROJECT_NAME}/${tenantOrEnvironment}`,
'tenant': tenantOrEnvironment.toUpperCase(),
});
});
console.dir(matrixJobArrayJson, { depth: null });
}
}
magnificent-soccer-44287
02/19/2024, 9:08 PMmagnificent-soccer-44287
02/19/2024, 9:09 PMgreat-sundown-78827
02/19/2024, 9:10 PMlittle-cartoon-10569
02/19/2024, 9:10 PM__main__.py
, or whatever your preferred language's default entrypoint.magnificent-soccer-44287
02/19/2024, 9:11 PMlittle-cartoon-10569
02/19/2024, 9:12 PMmagnificent-soccer-44287
02/19/2024, 9:13 PMlittle-cartoon-10569
02/19/2024, 9:14 PMmagnificent-soccer-44287
02/19/2024, 9:14 PMlittle-cartoon-10569
02/19/2024, 9:15 PMmagnificent-soccer-44287
02/19/2024, 9:16 PMmagnificent-soccer-44287
02/19/2024, 9:16 PMlittle-cartoon-10569
02/19/2024, 9:16 PMmagnificent-soccer-44287
02/19/2024, 9:16 PMmagnificent-soccer-44287
02/19/2024, 9:16 PMmagnificent-soccer-44287
02/19/2024, 9:17 PMlittle-cartoon-10569
02/19/2024, 9:18 PMmagnificent-soccer-44287
02/19/2024, 9:18 PMmagnificent-soccer-44287
02/19/2024, 9:19 PMmagnificent-soccer-44287
02/19/2024, 9:20 PMmagnificent-soccer-44287
02/19/2024, 9:20 PMlittle-cartoon-10569
02/19/2024, 9:22 PMmagnificent-soccer-44287
02/19/2024, 9:22 PMmagnificent-soccer-44287
02/19/2024, 9:23 PMmagnificent-soccer-44287
02/19/2024, 9:24 PMlittle-cartoon-10569
02/19/2024, 9:24 PMmagnificent-soccer-44287
02/19/2024, 9:24 PMgreat-sundown-78827
02/19/2024, 9:43 PMconst stack = new Stack(app, 'MyStackName')
new Bucket(stack, 'MyBucket')
And that’s really it … you then run cdk ls
and you see MyStackName
.. and cdk deploy MyStackName
will deploy it. No state management, no pre-creation of a stack via a CLI … everything is in code. In fact, we regularly define new stacks just by editing a list or map in a typescript file in the Github UI, and creating a PR …
Now Pulumi…
I am trying to map this to how Stacks work in Pulumi. I get that Pulumi needs to maintain state … so sure, we populate an S3 bucket for that. I also get that to really deploy a Stack, it needs to write actual state into the bucket.. fine.
For developing though.. it feels like I am missing something. I want something akin to the new Stack(…)
concept, where we define a Stack
as a typescript class, then add various Pulumi components into the stack to define what is managed in that stack. Is there some way of writing Pulumi code in this way, where we can define these top level contructs like a PulumiStack
and then place objects onto that stack?
all without calling the CLI to pre-create some YAML file… that frankly doesnt seem to really matter that much?little-cartoon-10569
02/19/2024, 10:00 PMlittle-cartoon-10569
02/19/2024, 10:01 PMlittle-cartoon-10569
02/19/2024, 10:02 PMlittle-cartoon-10569
02/19/2024, 10:02 PMgreat-sundown-78827
02/19/2024, 10:07 PMStacks are instances of a project
… that is a significant distinction.. I think I want to touch on that for a sec. I imagined that Stacks
are a sub comonent of a Project
which is a component of an Organization
. So I figured a single Git repo might have several stacks that focus on different thigns… a PagerdutyStack
and a GrafanaStack
for example.
I hear you saying that a Stack
might be a Dev
version of projectA
and a Prod
version of projectA
.. but fundamentally the code is the same and roughly the resources they launch are the same.
Am I getting that right?great-sundown-78827
02/19/2024, 10:10 PMFooTeam
for “staging” vs “dev” vs “prod”.. its one team. We might create multiple Services (eg Foo (Stg)
, Foo (Prd)
) - but we don’t really have a need or reason to have say two different deployment pipelines for that. So given that model… it sounds like we might just equate a Project == Stack
in that case.little-cartoon-10569
02/19/2024, 10:23 PMlittle-cartoon-10569
02/19/2024, 10:24 PMlittle-cartoon-10569
02/19/2024, 10:26 PMgreat-sundown-78827
02/19/2024, 10:28 PMStack
was important for my mental model.little-cartoon-10569
02/19/2024, 10:28 PMgreat-sundown-78827
02/19/2024, 10:29 PM/** @format */
import { Component } from 'projen';
import { InlineProgramArgs, LocalWorkspace, PulumiFn } from '@pulumi/pulumi/automation';
import { PulumiProject } from './pulumi-project';
export interface PulumiStackOptions {
readonly secretsProvider: string;
readonly program: PulumiFn;
}
export class PulumiStack extends Component {
public readonly stackName: string;
public readonly projectName: string;
public organization: string = 'organization';
protected readonly secretsProvider: string;
protected readonly program: PulumiFn;
constructor(project: PulumiProject, id: string, options: PulumiStackOptions) {
super(project, id);
this.stackName = id;
this.projectName = project.name;
this.secretsProvider = options.secretsProvider;
this.program = options.program;
}
synthesize(): void {
this.asyncSynth().catch((err) => {
console.error('ERROR DURING SYNTH');
console.log(err);
throw err;
});
}
async asyncSynth(): Promise<void> {
const args: InlineProgramArgs = {
stackName: this.stackName,
projectName: this.projectName,
program: this.program,
};
const stack = await LocalWorkspace.createOrSelectStack(args, {
workDir: '.',
});
await stack.workspace.saveStackSettings(`${this.organization}/${this.projectName}/${this.stackName}`, {
secretsProvider: this.secretsProvider,
});
}
}
great-sundown-78827
02/19/2024, 10:30 PMnpx projen
which sets up the rest of the repo, would also set up the Pulumi.xxx.yaml
contents, as well as ultimately handle creation of the github deploy pipelinegreat-sundown-78827
02/19/2024, 10:36 PMPulumi.yaml
and Pulumi.$stackName.yaml
files…