When trying to replace a gcp sercet version I hit ...
# google-cloud
When trying to replace a gcp sercet version I hit the following error
Copy code
gcp:secretmanager:SecretVersion (otp-secret-version):
    error: deleting urn:pulumi:dev::trip-service::gcp:secretmanager/secretVersion:SecretVersion::otp-secret-version: 1 error occurred:
        * Error when reading or editing SecretVersion: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See <https://developers.google.com/identity/sign-in/web/devconsole-project>.
        "@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
        "domain": "<http://googleapis.com|googleapis.com>",
        "metadata": {
          "email": "<mailto:pulumi@ride-app-dev-2.iam.gserviceaccount.com|pulumi@ride-app-dev-2.iam.gserviceaccount.com>",
          "method": "google.cloud.secretmanager.v1.SecretManagerService.DestroySecretVersion",
          "service": "<http://secretmanager.googleapis.com|secretmanager.googleapis.com>"
        "reason": "ACCOUNT_STATE_INVALID"
I’m using pulumi ESC and OIDC
is your service account authorized for the secret manager?
Yep it has secret manager admin role
is the service account enabled / turned on? The error looks like the account state is bad... I'm guessing that's the service account state
you should be able to quickly tell by
glcoud iam service-accounts list
and then
gcloud iam service-accounts describe <some-service-account>
The service account should be good because it's used for all deployments through pulumi and every other stacks work fine
hmm, not sure then. I'm also fairly new to glcoud... I would try this next: • impersonate the service account • try performing the actions with gcloud (impersonating the service account) Hopefully that investigation leads you somewhere
When using Pulumi esc it automatically impersonates the service account. And the same happens
This is happening both in deployment and and cli using ESC
and it's just a single gcp project or stack? How do you have your stacks / projects set up?
For each project there are two stacks. Dev and prod. For this project this is currently happening only for dev stack because I don't want to test it on prod
and it's the same account across all projects?
Both dev and prod use different gcp account but they have same configuration because they are also managed through pulumi
So both projects are essentially mirror copies
@colossal-tailor-72573 I think i’ve found the issue. It’s not just secret manager operations but any operation at all. The email in the error is pulumi@ride-app-dev-2.iam.gserviceaccount.com but it should be pulumi@<project-number>.iam.gserviceaccount.com. I’m not impoersonatinve the svc acc on cli and ESC is configured correctly to use the correct email. idk why it’s happening
keep digging 🙂
@colossal-tailor-72573 after a lot of digging, still can’t figure out why
Looks like somehow email Is being hard linked to the stack. Even if for dev stack I change the environment from app-dev to app-prod or for prod stack do the opposite it still sticks to that email. @colossal-tailor-72573 by any chance could it be a bug in the internal representation of pulumi state?
It is trying to use a deleted service account instead of the one defined in ESC. @plain-diamond-92898 can you help out here?
Checking with the team. (Apologies for the delay in response. Was out of office last few days)
@colossal-quill-8119 - Were you able to figure it out?
it still has the email hard linked
Okay. To confirm, GCP OIDC is working for you, and Pull secrets from the GCP secret manager is working? You are facing an issue when you are trying to write a secret to GCP using the credential you obtained via OIDC?
OIDC with ESC is working fine for EVERY OTHER stack other than this
No operation in this stack works. Not only secrets
Pulumi up/refresh doesn't work either, correct? can you share your ESC definition, pulumi config file for a stack that's working, and for this stack that doesn't work. If you could also capture a video that'd be really helpful!
Sure. Give me half an hour and I'll send you a loom on your dm?
That works! Thank you.