https://pulumi.com
Join Slack
Hi folks, I'm getting the following error anytime I pass in a provider to my awsx.VPC: ``` error...
# typescript
b
Hi folks, I'm getting the following error anytime I pass in a provider to my awsx.VPC:
Copy code
error: awsx:ec2:Vpc resource 'enterprise-dev-main-vpc' has a problem: Cannot read properties of undefined (reading 'length')
    error: awsx:ec2:Vpc resource 'shared-dev-main-vpc' has a problem: Cannot read properties of undefined (reading 'length')
    error: TypeError: Cannot read properties of undefined (reading 'length')
        at Vpc.getDefaultAzs (/snapshot/awsx/bin/ec2/vpc.js:285:26)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at Vpc.initialize (/snapshot/awsx/bin/ec2/vpc.js:51:103)
    error: TypeError: Cannot read properties of undefined (reading 'length')
        at Vpc.getDefaultAzs (/snapshot/awsx/bin/ec2/vpc.js:285:26)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at Vpc.initialize (/snapshot/awsx/bin/ec2/vpc.js:51:103)
I'll leave some code in the threads to prevent clogging the main channel, but, for debugging reasons, when I isolate what is actually causing issues, it's only when I remove my provider that things start working. I'm using AWS OUs for my multi-tenanted architecture and created a parent organization with SCPs. The SCP then uses a Provider to create a VPC in an OU.
Copy code
// Step 1: Create the AWS Organizations structure
const orgStructure = createOrganizationsStructure("<mailto:aws-tools@chaeth.com|aws-tools@chaeth.com>")

// Step 2: Create and attach Service Control Policies
const policies = createServiceControlPolicies()
attachSCPsToOUs(policies, orgStructure.toolsOU, orgStructure.tenantsParentOU)

// Step 3: Create provider for Tools account
const toolsProvider = new aws.Provider("tools-provider", {
  region: region,
  assumeRoles: [{
    roleArn: orgStructure.toolsAccount.id.apply(
      id => `arn:aws:iam::${id}:role/OrganizationAccountAccessRole`
    ),
  }],
})

// Step 4: Deploy central resources in Tools account
const toolsResources = createCentralDeployer(
  "ChaethAI",
  "LibreChat",
  undefined,
  toolsProvider
)

// Step 5: Create tenant accounts and deploy infrastructure
const tenantOutputs: Record<string, any> = {}

for (const tenant of tenants) {
  // Create tenant OU and account
  const tenantAccountInfo = createTenantOUAndAccount(
    tenant,
    orgStructure.tenantsParentOU.id,
  )

  // Create provider for this tenant account
  const tenantProvider = new aws.Provider(`${tenant.tenantId}-provider`, {
    region: region,
    assumeRoles: [
      {
        roleArn: tenantAccountInfo.tenantAccount.id.apply(
          id => `arn:aws:iam::${id}:role/OrganizationAccountAccessRole`
        ),
      }
    ],
  }, {
    dependsOn: [tenantAccountInfo.tenantAccount],
  })

  // Deploy tenant IAM resources
  const tenantIAMResources = createTenantIAMResources(
    tenant.tenantId,
    orgStructure.toolsAccount.id,
    tenantAccountInfo.tenantOU,
    tenantProvider
  )

  // Deploy tenant infrastructure
  const tenantInfra = new TenantInfrastructure(
    `${tenant.tenantId}-infra`,
    {
      tenant: {
        ...tenant,
        region: region,
      },
      environment: stackName,
      sharedConfig: {
        ...sharedConfig,
        libreChatEcrUrl: toolsResources.ecrRepository.repositoryUrl,
      },
    },
    {
      parent: tenantAccountInfo.tenantOU,
      provider: tenantProvider,
      dependsOn: [tenantAccountInfo.tenantOU, tenantProvider],
    }
  )
export function createOrganizationsStructure(
  toolsAccountEmail: string,
  parent?: pulumi.ComponentResource,
): OrganizationsOutputs {
  // Create or get the organization (this should already exist)
  const organization = new aws.organizations.Organization(
    "chaeth",
    {
      awsServiceAccessPrincipals: [
        "<http://cloudtrail.amazonaws.com|cloudtrail.amazonaws.com>",
        "<http://config.amazonaws.com|config.amazonaws.com>",
        "<http://sso.amazonaws.com|sso.amazonaws.com>",
      ],
      featureSet: "ALL",
    },
    { parent },
  )

  const rootId = organization.roots[0].id

  // Create Tools OU for centralized resources
  const toolsOU = new aws.organizations.OrganizationalUnit(
    "tools-ou",
    {
      name: "Tools",
      parentId: rootId,
      tags: {
        Name: "Tools OU",
        Purpose: "Centralized tools and ECR",
        ManagedBy: "Pulumi",
      },
    },
    { parent, dependsOn: [organization] },
  )

  // Create Tools account
  const toolsAccount = new aws.organizations.Account(
    "tools-account",
    {
      name: "Chaeth Tools",
      email: toolsAccountEmail,
      parentId: toolsOU.id,
      roleName: "OrganizationAccountAccessRole",
      tags: {
        Name: "Tools Account",
        Purpose: "Central ECR and deployment tools",
        ManagedBy: "Pulumi",
      },
    },
    {
      parent,
      dependsOn: [toolsOU],
      customTimeouts: {
        create: "20m",
      },
    },
  )

  // Create Tenants OU as parent for all tenant OUs
  const tenantsParentOU = new aws.organizations.OrganizationalUnit(
    "tenants-parent-ou",
    {
      name: "Tenants",
      parentId: rootId,
      tags: {
        Name: "Tenants Parent OU",
        Purpose: "Parent OU for all tenant OUs",
        ManagedBy: "Pulumi",
      },
    },
    { parent, dependsOn: [organization] },
  )

  return {
    organization,
    rootId,
    toolsOU,
    toolsAccount,
    tenantsParentOU,
  }
}
export function attachSCPsToOUs(
  policies: { [key: string]: aws.organizations.Policy },
  toolsOU: aws.organizations.OrganizationalUnit,
  tenantsParentOU: aws.organizations.OrganizationalUnit,
  parent?: pulumi.ComponentResource,
): void {
  // Attach tools policy to tools OU
  new aws.organizations.PolicyAttachment(
    "tools-scp-attachment",
    {
      policyId: policies.tools.id,
      targetId: toolsOU.id,
    },
    { parent },
  )

  // Attach tenant policy to tenants parent OU (will inherit to all tenant OUs)
  new aws.organizations.PolicyAttachment(
    "tenant-scp-attachment",
    {
      policyId: policies.tenant.id,
      targetId: tenantsParentOU.id,
    },
    { parent },
  )
}
Copy code
export function createTenantOUAndAccount(
  tenant: TenantConfig,
  tenantsParentOUId: pulumi.Output<string>,
  parent?: pulumi.ComponentResource,
): TenantAccountOutputs {
  // Create OU for the tenant
  const tenantOU = new aws.organizations.OrganizationalUnit(
    `${tenant.tenantId}-ou`,
    {
      name: `Tenant-${tenant.tenantId}`,
      parentId: tenantsParentOUId,
      tags: {
        Name: `${tenant.tenantId} OU`,
        TenantId: tenant.tenantId,
        TenantName: tenant.tenantName,
        ManagedBy: "Pulumi",
      },
    },
    { parent },
  )

  // Create AWS account for the tenant
  const accountEmail = `aws-tenant+${tenant.tenantId}@chaeth.com`
  const tenantAccount = new aws.organizations.Account(
    `${tenant.tenantId}-account`,
    {
      name: `${tenant.tenantName} (${tenant.tenantId})`,
      email: accountEmail,
      parentId: tenantOU.id,
      roleName: "OrganizationAccountAccessRole",
      tags: {
        TenantId: tenant.tenantId,
        TenantName: tenant.tenantName,
        Environment: "prod",
        ManagedBy: "Pulumi",
      },
    },
    {
      parent,
      dependsOn: [tenantOU],
      customTimeouts: {
        create: "20m",
      },
    },
  )

  return {
    tenantOU,
    tenantAccount,
  }
}

export interface TenantIAMOutputs {
  tenantDeployerRole: aws.iam.Role
  tenantExecutionRole: aws.iam.Role
  tenantTaskRole: aws.iam.Role
}

/**
 * Creates IAM resources within a tenant account for cross-account deployment
 */
export function createTenantIAMResources(
  tenantId: string,
  toolsAccountId: pulumi.Output<string>,
  parent?: pulumi.Resource,
  provider?: pulumi.ProviderResource,
): TenantIAMOutputs {
  const providerOpts = provider ? { provider } : {}

  // ========================================
  // TenantDeployer Role (for CI/CD)
  // ========================================

  // Trust policy: Only CentralDeployer from tools account can assume
  const tenantDeployerAssumePolicy = aws.iam.getPolicyDocumentOutput({
    statements: [
      {
        effect: "Allow",
        principals: [
          {
            type: "AWS",
            identifiers: [toolsAccountId.apply(id => `arn:aws:iam::${id}:role/CentralDeployer`)],
          },
        ],
        actions: ["sts:AssumeRole"],
        conditions: [
          {
            test: "StringEquals",
            variable: "sts:ExternalId",
            values: ["chaeth-ci"],
          },
        ],
      },
    ],
  })

  const tenantDeployerRole = new aws.iam.Role(
    `${tenantId}-TenantDeployer`,
    {
      name: "TenantDeployer",
      assumeRolePolicy: tenantDeployerAssumePolicy.json,
      tags: {
        Name: "TenantDeployer",
        TenantId: tenantId,
        Purpose: "CrossAccountDeployment",
        ManagedBy: "Pulumi",
      },
    },
    { parent, ...providerOpts },
  )

  // Full admin permissions for deployment
  new aws.iam.RolePolicyAttachment(
    `${tenantId}-TenantDeployer-AdminPolicy`,
    {
      role: tenantDeployerRole.name,
      policyArn: "arn:aws:iam::aws:policy/AdministratorAccess",
    },
    { parent, ...providerOpts },
  )

  // ========================================
  // TenantExecutionRole (for ECS tasks)
  // ========================================

  const tenantExecutionRole = new aws.iam.Role(
    `${tenantId}-TenantExecutionRole`,
    {
      name: "TenantExecutionRole",
      assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [
          {
            Action: "sts:AssumeRole",
            Effect: "Allow",
            Principal: {
              Service: "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>",
            },
          },
        ],
      }),
      tags: {
        Name: "TenantExecutionRole",
        TenantId: tenantId,
        Purpose: "ECSTaskExecution",
        ManagedBy: "Pulumi",
      },
    },
    { parent, ...providerOpts },
  )

  // Attach AWS managed policy for ECS task execution
  new aws.iam.RolePolicyAttachment(
    `${tenantId}-TenantExecutionRole-EcsPolicy`,
    {
      role: tenantExecutionRole.name,
      policyArn: "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
    },
    { parent, ...providerOpts },
  )

  // Add ECR pull permissions for central repository
  const executionRoleEcrPolicy = new aws.iam.Policy(
    `${tenantId}-TenantExecutionRoleEcrPolicy`,
    {
      name: "TenantExecutionRoleEcrPolicy",
      policy: aws.iam
        .getPolicyDocumentOutput({
          statements: [
            {
              sid: "ECRCrossAccountPull",
              effect: "Allow",
              actions: [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken",
              ],
              resources: [
                "*", // Authorization token requires *
                toolsAccountId.apply(id => `arn:aws:ecr:*:${id}:repository/librechat`),
              ],
            },
          ],
        })
        .json,
      tags: {
        Name: "TenantExecutionRoleEcrPolicy",
        TenantId: tenantId,
        ManagedBy: "Pulumi",
      },
    },
    { parent, ...providerOpts },
  )

  new aws.iam.RolePolicyAttachment(
    `${tenantId}-TenantExecutionRole-EcrPolicy`,
    {
      role: tenantExecutionRole.name,
      policyArn: executionRoleEcrPolicy.arn,
    },
    { parent, ...providerOpts },
  )

  // ========================================
  // TenantTaskRole (for ECS task runtime)
  // ========================================

  const tenantTaskRole = new aws.iam.Role(
    `${tenantId}-TenantTaskRole`,
    {
      name: "TenantTaskRole",
      assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [
          {
            Action: "sts:AssumeRole",
            Effect: "Allow",
            Principal: {
              Service: "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>",
            },
          },
        ],
      }),
      tags: {
        Name: "TenantTaskRole",
        TenantId: tenantId,
        Purpose: "ECSTaskRuntime",
        ManagedBy: "Pulumi",
      },
    },
    { parent, ...providerOpts },
  )

  // Task role permissions (Secrets Manager, Parameter Store, etc.)
  const taskRolePolicy = new aws.iam.Policy(
    `${tenantId}-TenantTaskRolePolicy`,
    {
      name: "TenantTaskRolePolicy",
      policy: aws.iam
        .getPolicyDocumentOutput({
          statements: [
            {
              sid: "ParameterStore",
              effect: "Allow",
              actions: [
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:GetParametersByPath",
              ],
              resources: [`arn:aws:ssm:*:*:parameter/*/${tenantId}/*`],
            },
            {
              sid: "SecretsManager",
              effect: "Allow",
              actions: ["secretsmanager:GetSecretValue"],
              resources: [`arn:aws:secretsmanager:*:*:secret:*`],
            },
            {
              sid: "CloudWatchLogs",
              effect: "Allow",
              actions: [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
              ],
              resources: ["*"],
            },
            {
              sid: "S3Access",
              effect: "Allow",
              actions: [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
              ],
              resources: [
                `arn:aws:s3:::*`,
                `arn:aws:s3:::*/*`,
              ],
            },
          ],
        })
        .json,
      tags: {
        Name: "TenantTaskRolePolicy",
        TenantId: tenantId,
        ManagedBy: "Pulumi",
      },
    },
    { parent, ...providerOpts },
  )

  new aws.iam.RolePolicyAttachment(
    `${tenantId}-TenantTaskRole-Policy`,
    {
      role: tenantTaskRole.name,
      policyArn: taskRolePolicy.arn,
    },
    { parent, ...providerOpts },
  )

  return {
    tenantDeployerRole,
    tenantExecutionRole,
    tenantTaskRole,
  }
}
Copy code
export class TenantInfrastructure extends pulumi.ComponentResource {
  public readonly outputs: TenantInfrastructureOutputs

  constructor(
    name: string,
    args: {
      tenant: TenantConfig
      environment: string
      sharedConfig: SharedInfraConfig
    },
    opts?: pulumi.ComponentResourceOptions,
  ) {
    super("chaeth:infrastructure:Tenant", name, {}, opts)

    const { tenant, environment, sharedConfig } = args

    const vpc = createTenantVpc(tenant, environment, this)
export function createTenantVpc(
  tenant: TenantConfig,
  environment: string,
  parent?: pulumi.ComponentResource,
): VpcOutputs {
  const namePrefix = `${tenant.tenantId}-${environment}`

  const vpc = new awsx.ec2.Vpc(
    `${namePrefix}-main-vpc`,
    {
      cidrBlock: "10.0.0.0/16",
      numberOfAvailabilityZones: 2,
      enableDnsHostnames: true,
      enableDnsSupport: true,
      natGateways: {
        strategy:
          environment === "prod"
            ? awsx.ec2.NatGatewayStrategy.OnePerAz
            : awsx.ec2.NatGatewayStrategy.Single,
      },
      tags: {
        ...tenant.tags,
        Name: "main-vpc",
        TenantId: tenant.tenantId,
        Environment: environment,
        ManagedBy: "Pulumi",
      },
    },
    { parent },
  )
shit i think i forgot to opt into my region
l
Did that resolve it? There's a lot of code to read, but I'll have a go if it's still broken! 🙂 BTW You should have a look at the "Text Snippet" feature of Slack, in the "+" button of the text entry box. It makes a collapsible syntax-highlighted text box for your code. It's lovely.
🙌 1
b
Possibly @little-cartoon-10569! I went ahead and created a new AWS account and enabled all the regions manually in the UI. I won't know if the VPC issue is resolved until I resolve a top-level await issue in Pulumi. Specifically, it seems like I had to first check that an account was created prior to assuming a role via 
aws.Provider
The only way to check if the account was created was with
Copy code
const toolsAccount = await awsNative.organizations.getAccount({
  accountId: orgStructure.toolsAccount.id.get()
})
if (toolsAccount.status === "ACTIVE") { /* Perform VPC creation */ }
Unfortunately, I'm now blocked on figuring out how to configure the await (https://github.com/pulumi/pulumi/issues/5161) and nothing seems to work haha
l
IMO this sort of thing should happen outside your Pulumi project. I think you might benefit from automation-api here, since the top-level program can check for accounts and stuff like that, and then call the Pulumi project (which can be inline if you like) once your preconditions are met.
👍 1
Essentially, wrap your declarative projects within multi-step imperative programs.
b
unfortunately, still getting the VPC error:/
l
I'll see if I can find the code for 
getDefaultAzs()
.
You're using new awsx, right? Not classic?
👍 1
Looks like the compile line might map to this TS line:
Copy code
const result = await aws.getAvailabilityZones(undefined, { parent: this });
if (result.names.length < desiredCount) {
  ...
Checking in the AWS SDK what result.names is supposed to contain when 
getAvailabilityZones(undefined)
is called.
It just calls 
aws:index/getAvailabilityZones:getAvailabilityZones
with an empty arg object. Which I presume is the same as calling 
aws ec2 describe-availability-zones
? What happens if you run that using the same profile / creds as Pulumi is using?
b
yes i think it has to do with the provider, using OUs, and using a separate account per OU in a multi-tenanted environment. My pulumi code also enables the region it'll be located in (I think aws ap-southeast-7 requires opt in on a per account basis). My code uses OrganizationAccountAccessRole. It's possible that, during preview, the region isn't enabled yet causing the crash, so i'll need to do some multi step thing again which is a bit annoying since I don't think dependsOn will help here
i think the easiest short term solution is to separate the enabling of the region and creation of resources in the account into separate deployment steps, but I feel like this is going to get annoying. a possible better alternative would be to run a pulumi stack per tenant, but at that point I feel like I would go back to using Terraform/Terragrunt
l
You should absolutely use more projects and arrange them correctly. That was a big learn for me early on: putting everything in one project is rarely a good idea.
👍 1
VPC peering is where I learned that lesson. One project for peered VPCs isn't good.
b
gotcha
how did you end up structuring VPC peering if it required knowing the outputs of two VPCs?
l
In my case, all the VPCs are identical, according to our standards. So 1 project has a stack for each VPC: emea-prod, apac-preprod, etc. And one extra stack for our central VPC, called deployment. In a different project, we have another stack with the same name as in the first project, and this project's job is just to set up peering between the existing two VPCs, deployment and <current-stack>.
So both VPCs exist during before peering, and the peering project has two providers: the deployment one, and the stack-specific one.
Stack references sort all the rest.
👍 1
12 Views
Open in Slack
PreviousNext
Powered by

Pulumi Community

No matter how you like to participate in developer communities, Pulumi wants to meet you there. If you want to meet other Pulumi users to share use-cases and best practices, contribute code or documentation, see us at an event, or just tell a story about something cool you did with Pulumi, you are part of our community.

Powered by