https://pulumi.com logo
#google-cloud
Title
# google-cloud
c

colossal-quill-8119

12/29/2023, 6:51 PM
When trying to replace a gcp sercet version I hit the following error
Copy code
gcp:secretmanager:SecretVersion (otp-secret-version):
    error: deleting urn:pulumi:dev::trip-service::gcp:secretmanager/secretVersion:SecretVersion::otp-secret-version: 1 error occurred:
        * Error when reading or editing SecretVersion: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See <https://developers.google.com/identity/sign-in/web/devconsole-project>.
    Details:
    [
      {
        "@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
        "domain": "<http://googleapis.com|googleapis.com>",
        "metadata": {
          "email": "<mailto:pulumi@ride-app-dev-2.iam.gserviceaccount.com|pulumi@ride-app-dev-2.iam.gserviceaccount.com>",
          "method": "google.cloud.secretmanager.v1.SecretManagerService.DestroySecretVersion",
          "service": "<http://secretmanager.googleapis.com|secretmanager.googleapis.com>"
        },
        "reason": "ACCOUNT_STATE_INVALID"
      }
    ]
I’m using pulumi ESC and OIDC
c

colossal-tailor-72573

12/30/2023, 4:29 AM
is your service account authorized for the secret manager?
c

colossal-quill-8119

12/30/2023, 4:29 AM
Yep it has secret manager admin role
c

colossal-tailor-72573

12/30/2023, 4:31 AM
is the service account enabled / turned on? The error looks like the account state is bad... I'm guessing that's the service account state
you should be able to quickly tell by
glcoud iam service-accounts list
and then
gcloud iam service-accounts describe <some-service-account>
c

colossal-quill-8119

12/30/2023, 4:34 AM
The service account should be good because it's used for all deployments through pulumi and every other stacks work fine
c

colossal-tailor-72573

12/30/2023, 4:38 AM
hmm, not sure then. I'm also fairly new to glcoud... I would try this next: • impersonate the service account • try performing the actions with gcloud (impersonating the service account) Hopefully that investigation leads you somewhere
c

colossal-quill-8119

12/30/2023, 4:40 AM
When using Pulumi esc it automatically impersonates the service account. And the same happens
This is happening both in deployment and and cli using ESC
c

colossal-tailor-72573

12/30/2023, 5:09 AM
and it's just a single gcp project or stack? How do you have your stacks / projects set up?
c

colossal-quill-8119

12/30/2023, 5:10 AM
For each project there are two stacks. Dev and prod. For this project this is currently happening only for dev stack because I don't want to test it on prod
c

colossal-tailor-72573

12/30/2023, 5:11 AM
and it's the same account across all projects?
c

colossal-quill-8119

12/30/2023, 5:12 AM
Both dev and prod use different gcp account but they have same configuration because they are also managed through pulumi
So both projects are essentially mirror copies
@colossal-tailor-72573 I think i’ve found the issue. It’s not just secret manager operations but any operation at all. The email in the error is pulumi@ride-app-dev-2.iam.gserviceaccount.com but it should be pulumi@<project-number>.iam.gserviceaccount.com. I’m not impoersonatinve the svc acc on cli and ESC is configured correctly to use the correct email. idk why it’s happening
c

colossal-tailor-72573

12/30/2023, 4:55 PM
keep digging 🙂
c

colossal-quill-8119

01/25/2024, 7:56 AM
@colossal-tailor-72573 after a lot of digging, still can’t figure out why
c

colossal-tailor-72573

01/26/2024, 5:22 AM
😞
c

colossal-quill-8119

01/26/2024, 6:09 AM
Looks like somehow email Is being hard linked to the stack. Even if for dev stack I change the environment from app-dev to app-prod or for prod stack do the opposite it still sticks to that email. @colossal-tailor-72573 by any chance could it be a bug in the internal representation of pulumi state?
It is trying to use a deleted service account instead of the one defined in ESC. @plain-diamond-92898 can you help out here?
p

plain-diamond-92898

01/31/2024, 8:50 PM
Checking with the team. (Apologies for the delay in response. Was out of office last few days)
@colossal-quill-8119 - Were you able to figure it out?
c

colossal-quill-8119

02/14/2024, 3:22 PM
nope
it still has the email hard linked
p

plain-diamond-92898

02/15/2024, 3:42 PM
Okay. To confirm, GCP OIDC is working for you, and Pull secrets from the GCP secret manager is working? You are facing an issue when you are trying to write a secret to GCP using the credential you obtained via OIDC?
c

colossal-quill-8119

02/15/2024, 3:43 PM
OIDC with ESC is working fine for EVERY OTHER stack other than this
No operation in this stack works. Not only secrets
p

plain-diamond-92898

02/15/2024, 3:55 PM
Pulumi up/refresh doesn't work either, correct? can you share your ESC definition, pulumi config file for a stack that's working, and for this stack that doesn't work. If you could also capture a video that'd be really helpful!
c

colossal-quill-8119

02/15/2024, 3:56 PM
Sure. Give me half an hour and I'll send you a loom on your dm?
p

plain-diamond-92898

02/15/2024, 3:57 PM
That works! Thank you.
c

colossal-quill-8119

02/15/2024, 4:27 PM
sent