https://pulumi.com logo
Join Slack
Powered by
At our company our pulumi stack has been managed b...
# general
g
At our company our pulumi stack has been managed by a single dev since we started using pulumi. We have not hired on a new engineer to work in the pulumi stack. Unfortunatley this now has created a problem. Situation: Aws lambdas written in go, when we run pulumi up on engineer 1s computer (who originally deployed everything) there are no changes. But then when engineer 2 does pulumi up (no code changes) it says the last modified and code has changed (they had to seperartley build the bootstrap executable) Are there any solutions to this?
Copy code
.PHONY: build clean deploy

build:
	GOOS=linux GOARCH=arm64 CGO_ENABLED=0 \
	SOURCE_DATE_EPOCH=0 \
	go build -trimpath -buildvcs=false -ldflags="-buildid= -s -w" -tags=lambda.norpc -o bootstrap .

clean:
	rm -f bootstrap

deploy: clean build
this is my make command
m
So Pulumi believes that there has been a change to the Lambda's source code and wants to update the Lambda?
s
Can you post the code of the 
aws.lambda.Function
resource as well?
g
Copy code
// biome-ignore lint/performance/noNamespaceImport: node convention
import * as path from 'node:path'
// biome-ignore lint/performance/noNamespaceImport: pulumi convention
import * as aws from '@pulumi/aws'
// biome-ignore lint/performance/noNamespaceImport: pulumi convention
import * as pulumi from '@pulumi/pulumi'

export type HttpMethod = 'GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH'

const DEFAULT_MEMORY_SIZE_MB = 256
const DEFAULT_TIMEOUT_SECONDS = 10
const MB_TO_KB = 1024

export interface LambdaHttpRouteArgs {
  httpApi: aws.apigatewayv2.Api
  routePath: string
  methods: HttpMethod[]
  handlerPath: string
  runtime: aws.lambda.Runtime
  handler: string
  architecture?: 'x86_64' | 'arm64'
  subnetIds: pulumi.Input<string[]>
  securityGroupIds: pulumi.Input<string[]>
  memorySizeMb?: number
  timeoutSeconds?: number
  existingRoleArn: pulumi.Input<string>
  authorizerId?: pulumi.Input<string>
  environment?: Record<string, pulumi.Input<string>>
  layers?: pulumi.Input<string[]>
  ephemeralStorageGb?: number
  // Image package type options (alternative to Zip)
  packageType?: 'Zip' | 'Image'
  imageUri?: pulumi.Input<string>
}

export class LambdaHttpRoute extends pulumi.ComponentResource {
  readonly function: aws.lambda.Function
  readonly routes: aws.apigatewayv2.Route[]

  constructor(name: string, args: LambdaHttpRouteArgs, opts?: pulumi.ComponentResourceOptions) {
    super('pkg:foundation:api:LambdaHttpRoute', name, {}, opts)

    const packageType = args.packageType ?? 'Zip'
    if (packageType === 'Image' && !args.imageUri) {
      throw new Error('imageUri is required when packageType is Image')
    }

    const baseFunctionArgs = {
      role: args.existingRoleArn,
      architectures: args.architecture ? [args.architecture] : undefined,
      memorySize: args.memorySizeMb ?? DEFAULT_MEMORY_SIZE_MB,
      timeout: args.timeoutSeconds ?? DEFAULT_TIMEOUT_SECONDS,
      environment: args.environment ? { variables: args.environment } : undefined,
      vpcConfig: {
        subnetIds: args.subnetIds,
        securityGroupIds: args.securityGroupIds,
      },
      ephemeralStorage: args.ephemeralStorageGb
        ? { size: args.ephemeralStorageGb * MB_TO_KB }
        : undefined,
    }

    const functionArgs =
      packageType === 'Image'
        ? {
            ...baseFunctionArgs,
            packageType: 'Image' as const,
            imageUri: args.imageUri,
          }
        : {
            ...baseFunctionArgs,
            packageType: 'Zip' as const,
            runtime: args.runtime,
            handler: args.handler,
            code: new pulumi.asset.AssetArchive({
              // The key becomes the filename inside the Zip uploaded to Lambda
              bootstrap: new pulumi.asset.FileAsset(path.join(args.handlerPath, 'bootstrap')),
            }),
            layers: args.layers,
          }

    this.function = new aws.lambda.Function(`${name}-fn`, functionArgs, {
      parent: this,
    })

    const integration = new aws.apigatewayv2.Integration(
      `${name}-int`,
      {
        apiId: args.httpApi.id,
        integrationType: 'AWS_PROXY',
        integrationUri: this.function.arn,
        payloadFormatVersion: '2.0',
      },
      { parent: this }
    )

    this.routes = args.methods.map(
      (method) =>
        new aws.apigatewayv2.Route(
          `${name}-route-${method.toLowerCase()}`,
          {
            apiId: args.httpApi.id,
            routeKey: `${method} /${args.routePath}`,
            target: pulumi.interpolate`integrations/${integration.id}`,
            authorizationType: args.authorizerId ? 'JWT' : 'NONE',
            authorizerId: args.authorizerId,
          },
          { parent: this }
        )
    )

    const accountId = pulumi.output(aws.getCallerIdentity()).apply((i) => i.accountId)

    new aws.lambda.Permission(
      `${name}-invoke`,
      {
        action: 'lambda:InvokeFunction',
        function: this.function.name,
        principal: '<http://apigateway.amazonaws.com|apigateway.amazonaws.com>',
        sourceArn: pulumi.interpolate`arn:aws:execute-api:${aws.config.region}:${accountId}:${args.httpApi.id}/*/*/${args.routePath}`,
      },
      { parent: this }
    )

    this.registerOutputs({
      function: this.function,
      routes: this.routes,
    })
  }
}
these get created in a lambda factory:
Copy code
/** biome-ignore-all lint/performance/noNamespaceImport: pulumi said so */
import type * as aws from '@pulumi/aws'
import type * as pulumi from '@pulumi/pulumi'
import type { SharedHttpApi } from '../../constructs/foundation/api'
import { LambdaHttpRoute } from '../../constructs/foundation/api'

export type RoleType =
  | 'adminDbRead'
  | 'appDbRead'
  | 'appDbReadSqsSend'
  | 'adminS3List'
  | 'fileUpload'
  | 'fileProcessor'

export interface RouteDef {
  name: string
  path: string
  methods: ('GET' | 'POST' | 'PUT' | 'DELETE' | 'PATCH')[]
  handlerPath: string
  role: RoleType
  timeoutSeconds?: number
  memorySizeMb?: number
  env?: Record<string, pulumi.Input<string>>
}

interface RouteContext {
  api: SharedHttpApi
  authorizerId: pulumi.Input<string> | undefined
  vpc: {
    subnetIds: pulumi.Input<string[]>
  }
  securityGroupIds: pulumi.Input<string[]>
  roles: Partial<Record<RoleType, aws.iam.Role>>
  baseEnv: Record<string, pulumi.Input<string>>
}

const DEFAULT_TIMEOUT_SECONDS = 30

export function createRoutes(defs: RouteDef[], ctx: RouteContext) {
  return defs.map((d) => {
    const role = ctx.roles[d.role]
    if (!role) {
      throw new Error(`Role '${d.role}' not found in context for route '${d.name}'`)
    }

    return new LambdaHttpRoute(d.name, {
      httpApi: ctx.api.api,
      routePath: d.path,
      methods: d.methods,
      handlerPath: d.handlerPath,
      runtime: 'provided.al2023',
      architecture: 'arm64',
      handler: 'bootstrap',
      subnetIds: ctx.vpc.subnetIds,
      authorizerId: ctx.authorizerId,
      existingRoleArn: role.arn,
      environment: { ...ctx.baseEnv, ...(d.env ?? {}) },
      securityGroupIds: ctx.securityGroupIds,
      timeoutSeconds: d.timeoutSeconds ?? DEFAULT_TIMEOUT_SECONDS,
      memorySizeMb: d.memorySizeMb,
    })
  })
}
and an array of itmes like this:
Copy code
export const appRoutes: RouteDef[] = [
  {
    name: 'app-projects-get',
    path: 'projects',
    methods: ['GET'],
    handlerPath: '../app/lambda/app/projects/get',
    role: 'appDbRead',
  },......
get passed in
s
Can you post the diff on Developer 2's machine? 
pulumi preview --diff
g
Copy code
}
          - lastModified: "2025-10-22T02:32:45.000+0000"
        ~ aws:lambda/function:Function: (update)
            [id=app-parcels-autocomplete-get-fn-c23fa10]
            [urn=urn:pulumi:development::backend::pkg:foundation:api:LambdaHttpRoute$aws:lambda/function:Function::app-parcels-autocomplete-get-fn]
            [provider=urn:pulumi:development::backend::pulumi:providers:aws::default_7_8_0::4bd3c38a-dae2-4ba1-b341-71f829fdb3d5]
          ~ code        : archive(assets:c550b59->544514b) {
              ~ "bootstrap": asset(file:9534128->2058616) { ../app/lambda/app/parcels/autocomplete/get/bootstrap }
            }
          - lastModified: "2025-10-22T02:32:36.000+0000"
        ~ aws:lambda/function:Function: (update)
            [id=admin-data-parcels-id-coordinates-get-fn-b13c58c]
            [urn=urn:pulumi:development::backend::pkg:foundation:api:LambdaHttpRoute$aws:lambda/function:Function::admin-data-parcels-id-coordinates-get-fn]
            [provider=urn:pulumi:development::backend::pulumi:providers:aws::default_7_8_0::4bd3c38a-dae2-4ba1-b341-71f829fdb3d5]
          ~ code        : archive(assets:d9790c1->3614bc2) {
              ~ "bootstrap": asset(file:6164dd5->d44a053) { ../app/lambda/admin/data/parcels/$parcel_id/coordinates/get/bootstrap }
            }
          - lastModified: "2025-10-22T02:32:35.000+0000"
        ~ aws:lambda/function:Function: (update)
            [id=app-users-recent-projects-get-fn-32dbe31]
            [urn=urn:pulumi:development::backend::pkg:foundation:api:LambdaHttpRoute$aws:lambda/function:Function::app-users-recent-projects-get-fn]
            [provider=urn:pulumi:development::backend::pulumi:providers:aws::default_7_8_0::4bd3c38a-dae2-4ba1-b341-71f829fdb3d5]
          ~ code        : archive(assets:0085a6a->cab8fb0) {
              ~ "bootstrap": asset(file:2fbe49c->1902c19) { ../app/lambda/app/users/recent-projects/get/bootstrap }
            }
          - lastModified: "2025-10-22T02:32:48.000+0000"
        ~ aws:lambda/function:Function: (update)
            [id=app-projects-id-reports-id-get-fn-883be31]
            [urn=urn:pulumi:development::backend::pkg:foundation:api:LambdaHttpRoute$aws:lambda/function:Function::app-projects-id-reports-id-get-fn]
            [provider=urn:pulumi:development::backend::pulumi:providers:aws::default_7_8_0::4bd3c38a-dae2-4ba1-b341-71f829fdb3d5]
          ~ code        : archive(assets:e93ab7e->58da86c) {
              ~ "bootstrap": asset(file:008f72d->f8b3be1) { ../app/lambda/app/projects/{project_id}/reports/$report_id/get/bootstrap }
            }
          - lastModified: "2025-10-22T02:32:37.000+0000"
here is a snippet of it (its just this like 200 times
s
Yeah, the hash of that file clearly changed between the two development environments.
m
To me, this looks like your problem is indeed that the bootstrap executable is not identical, and thus the FileAsset (and, in turn, the AssetArchive) are different. So in that sense, it's correct that the Lambda is updated. I think the true solution to this problem is to make sure that the executable is identical by building it once and then distributing it either as a binary or inside a container image. This also has the benefit that you can unambiguously version your executable and can be sure that the particular binary you're running is functional. As a temporary workaround you can use the ignoreChanges resource option to ignore changes to 
bootstrap
or update only specific resources using the 
--target
option of the Pulumi CLI.
But maybe Josh knows some other tricks 😉
g
If i build it multiple times on one computer and run pulumi diff or pulumi up there are no changes, its only when its built on a different computer
m
I think getting reproducible builds on one machine is already difficult, and across machines it's even harder.
I've never had to worry about this with Go, but if you can build it reproducibly on one machine, you might be able to do that in some CI workflow. But then you're already so close to just building it once and storing it in ECR or S3 that you might as well just do that and get all the benefits of "publishing" binaries.
👍 1
☝️ 1
c
Looking for anyone who has used the Automation API with Pulumi, I got a Monorepo, which contains multiple applications and the Pulumi infrastructure code. I am making an application that will allow me to manage Pulumi stacks. My Infra State is stored in a S3 for context. The issue is that I would use RemoteWorkspace to create or select stacks, but then RemoteStack is super limited in its functionality compared to the standard Stack. But I need the ability to configure stacks, which RemoteStacks can't do.
m
What is your reason for using RemoteWorkspace over LocalWorkspace? My understanding is that RemoteWorkspace only works with Pulumi Deployments, which it sounds like you're not using.
c
Ah sorry, the terminology in https://www.pulumi.com/docs/iac/automation-api/concepts-terminology/ made it seem like all you need is the git repo to do it as the words "Pulumi Deployment" were not highlighted as a reference to something else.
Then my next question is if I deploy this a as containerised application, what would the flow be. As it says that it is basically a 1:1 Pulumi CLI for the most part, so does that mean I would need to push to the git repo everytime I make a change to stack config or settings of the project
m
No, you don't have to push to the Git repo. You need to pull the Pulumi program code into your local env, and then you run it. So it's really just like running the Pulumi CLI in a Docker container.
c
So I can just ignore the pulumi.[stack].yaml files
And just let the S3 do all the stack config storage
👍 1
m
Yes, you can entirely ignore them, you don't even need them. They might be created temporarily in the process.
c
Ah okay
Brilliant Thank you
Reviving this thread, I have discovered that the files are needed, for LocalWorkspace. https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/pulumi/classes/automation.LocalWorkspace.html So i need some help getting this to work again... The only thing i can think todo is to make a temp directory and to use Github Via HTTP to clone and make a merge a branch when i make changes, to be PR when everything is confirmed... Any Help would be great to make this easier.
m
Hmm, so I've worked with the Automation API before and I don't recall ever caring about the files. Can you describe your setup and why/where you believe you need them? I'm pretty sure you can make this work without handling the configuration files in the way you describe.
14 Views
Open in Slack
PreviousNext
Powered by