Has anyone successfully setup GCP OIDC with Pulumi...
# pulumi-deployments
c
Has anyone successfully setup GCP OIDC with Pulumi Cloud for deployments? I have followed all the docs multiple times to make sure i didn't miss anything. I also followed google cloud docs as well which did have some differences but no dice. Been banging my head against a wall for 3 days trying to get Pulumi cloud deployments working but I am about to give up and just do it all in github actions instead 😞
d
I haven’t on GCP—I’m using AWS, but what are you seeing?
c
Copy code
The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist.
I have tried soooo many difference audience values to get it to work, some i found in the pulumi docs but look like they could have been copy pasted from AWS docs some i pulled from google cloud docs, I have use various AI tools giving them the docs and code and all the context i could and tried all those suggestions. Nothing works 😭
I have verified the identity pool does indeed exist and is enabled, same for the identity pool provider
d
Copy code
import json

import pulumi
import pulumi_aws as aws

# Leave in legacy audience "myorg" for backwards compatibility
audiences = ["myorg", "aws:myorg"]

# Create OIDC provider for Pulumi Deployments
oidc_provider = aws.iam.OpenIdConnectProvider(
    "Pulumi OIDC Provider",
    client_id_lists=audiences,
    thumbprint_lists=["9e99a48a9960b14926bb7f3b02e22da2b0ab7280"],
    url="<https://api.pulumi.com/oidc>",
)

oidc_provider_role = aws.iam.Role(
    "Pulumi OIDC Provider Role",
    name="PulumiOIDC",
    assume_role_policy=oidc_provider.arn.apply(
        lambda arn: json.dumps(
            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Principal": {
                            "Federated": arn,
                        },
                        "Action": "sts:AssumeRoleWithWebIdentity",
                        "Condition": {
                            "StringEquals": {
                                "<http://api.pulumi.com/oidc:aud|api.pulumi.com/oidc:aud>": audiences,
                            }
                        },
                    }
                ],
            }
        )
    ),
    managed_policy_arns=["arn:aws:iam::aws:policy/AdministratorAccess"],
)

pulumi.export("PulumiOIDCRoleArn", oidc_provider_role.arn)
Is what I’m working with
I have a bootstrap stack I run manually to set that up
and then have deployments for all other stacks
and I use the exported value for the deployment config
Copy code
// Pulumi OIDC Configuration
const issuer = '<https://api.pulumi.com/oidc>';
const pulumiOrg = pulumi.getOrganization();

// Update the audience to use the correct format
const subIdPrefix = `pulumi:deploy:org:${pulumiOrg}:project:core:stack:${env}:operation`;
const subOperations = ['preview:scope:write', 'update:scope:write', 'refresh:scope:write', 'destroy:scope:write'];
const subIds = subOperations.map((operation) => `${subIdPrefix}:${operation}`);

// Create a Workload Identity Pool
const identityPool = new gcp.iam.WorkloadIdentityPool('pulumiOidcWorkloadIdentityPool', {
  workloadIdentityPoolId: `pl-oidc-id-pool-${env}`,
  description: 'Pulumi OIDC Workload Identity Pool',
  displayName: 'Pulumi OIDC Identity Pool',
});

// Create a Workload Identity Provider
const identityProvider = new gcp.iam.WorkloadIdentityPoolProvider('pulumiOidcIdentityProvider', {
  workloadIdentityPoolId: identityPool.workloadIdentityPoolId,
  workloadIdentityPoolProviderId: `pl-oidc-provider-${env}`,
  attributeMapping: {
    'google.subject': 'assertion.sub',
  },
  oidc: {
    issuerUri: issuer,
    allowedAudiences: [pulumiOrg],
  },
});

// Create a service account for OIDC
const oidcServiceAccount = new gcp.serviceaccount.Account('pulumiOidcServiceAccount', {
  accountId: `pl-oidc-sa-${env}`,
  displayName: 'Pulumi OIDC Service Account',
});

// Grant the service account editor role
const editorPolicyBinding = new gcp.projects.IAMMember('pulumiEditorIamBinding', {
  member: pulumi.interpolate`serviceAccount:${oidcServiceAccount.email}`,
  // TODO: Use a custom role instead of owner
  role: 'roles/owner',
  project: projectId,
});

const members = subIds.map(
  (subId) => pulumi.interpolate`<principal://iam.googleapis.com/${identityPool.name}/subject/${subId}>`
);

// Allow the workload identity pool to impersonate the service account
const iamPolicyBinding = new gcp.serviceaccount.IAMBinding('pulumiIamPolicyBinding', {
  serviceAccountId: oidcServiceAccount.name,
  role: 'roles/iam.workloadIdentityUser',
  members,
});

const settings = new pulumiService.DeploymentSettings('pulumiDeploymentSettings', {
  organization: pulumiOrg,
  project: repoName,
  stack: env,
  operationContext: {
    environmentVariables: {},
    oidc: {
      gcp: {
        projectId: projectId,
        providerId: identityProvider.workloadIdentityPoolProviderId,
        serviceAccount: oidcServiceAccount.email,
        tokenLifetime: '1h0m0s',
        workloadPoolId: identityPool.workloadIdentityPoolId,
      },
    },
  },
  sourceContext: {
    git: {
      branch: 'kaden/fix_pulumi_oidc',
      repoDir: 'pulumi',
    },
  },
  github: {
    repository: `OrgName/${repoName}`,
    previewPullRequests: true,
    deployCommits: true,
    paths: ['pulumi/**'],
  },
  cacheOptions: {
    enable: true,
  },
});
d
So it’s failing on trying to create the
new gcp.iam.WorkloadIdentityPoolProvider
resource when you up the stack?
c
No when trying to authenticate the updates to the cloud infra, e.g. update settings on my cloud sql instance
Here is the full error message
Copy code
pulumi:pulumi:Stack my-project-id running warning: resource plugin pulumiservice is expected to have version >=0.27.4, but has 0.0.1; the wrong version may be on your path, or this may be a bug in the plugin 
  ~  gcp:sql:DatabaseInstance corePostgresql updating (0s) [diff: ~settings] 
     gcp:sql:DatabaseInstance corePostgresql 
   error:   sdk-v2/provider2.go:515: sdk.helper_schema: Error when reading or editing SQL Database Instance "core-postgresql-staging-us-west3": Get "<https://sqladmin.googleapis.com/sql/v1beta4/projects/my-project-id/instances/core-postgresql-staging-us-west3?alt=json&prettyPrint=false>": oauth2/google: unable to generate access token: Post "<https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/pl-oidc-sa-staging@my-project-id.iam.gserviceaccount.com:generateAccessToken>": oauth2/google: status code 400: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}: provider=google-beta@8.16.0 
  ~  gcp:sql:DatabaseInstance corePostgresql updating (0s) [diff: ~settings]; error: 1 error occurred: 
  ~  gcp:sql:DatabaseInstance corePostgresql **updating failed** [diff: ~settings]; error: 1 error occurred: 
     pulumi:pulumi:Stack my-project-id running error: update failed 
     pulumi:pulumi:Stack my-project-id **failed** 1 error; 1 warning 
     gcp:sql:DatabaseInstance corePostgresql 
  **failed** 1 error 
 Diagnostics: 
   gcp:sql:DatabaseInstance (corePostgresql 
 ): 
     error:   sdk-v2/provider2.go:515: sdk.helper_schema: Error when reading or editing SQL Database Instance "core-postgresql-staging-us-west3": Get "<https://sqladmin.googleapis.com/sql/v1beta4/projects/my-project-id/instances/core-postgresql-staging-us-west3?alt=json&prettyPrint=false>": oauth2/google: unable to generate access token: Post "<https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/pl-oidc-sa-staging@my-project-id.iam.gserviceaccount.com:generateAccessToken>": oauth2/google: status code 400: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}: provider=google-beta@8.16.0
d
That’s from the deployment logs?
c
yeah the pulumi deployment logs
d
So the stack you posted above stands up without any errors. But when you try to run the deployment defined by that stack, it fails?
And the GCP project id is the same? Also—I’m not as familiar with GCP, but…—does the service account have permission(s) to use the pool?
c
Yeah I can run pulumi up from my computer and everything works. I also run it to setup the pulumi deployment oidc settings. So my testing involved creating a branch, pointing pulumi deployment settings for my stack to my branch using the
const settings = new pulumiService.DeploymentSettings('pulumiDeploymentSettings', {
and running that from my machine. Then making a change to the pulumi code to make a change to some cloud infra because previews always work no matter what the OIDC settings are. Then i push that to my branch and let pulumi deployments try run an update and thats when it fails
Well after some back and forth with their support team we figured it out!
The pulumi deployment config UI was asking for a GCP project ID but what it really wanted was the Project number which is a totally different thing 🤦 so some naming clashes led me astray
d
🤦🏽
glad you figured it out!