https://pulumi.com logo
Join Slack
Powered by
When trying to replace a gcp sercet version I hit ...
# google-cloud
c
When trying to replace a gcp sercet version I hit the following error
Copy code
gcp:secretmanager:SecretVersion (otp-secret-version):
    error: deleting urn:pulumi:dev::trip-service::gcp:secretmanager/secretVersion:SecretVersion::otp-secret-version: 1 error occurred:
        * Error when reading or editing SecretVersion: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See <https://developers.google.com/identity/sign-in/web/devconsole-project>.
    Details:
    [
      {
        "@type": "<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>",
        "domain": "<http://googleapis.com|googleapis.com>",
        "metadata": {
          "email": "<mailto:pulumi@ride-app-dev-2.iam.gserviceaccount.com|pulumi@ride-app-dev-2.iam.gserviceaccount.com>",
          "method": "google.cloud.secretmanager.v1.SecretManagerService.DestroySecretVersion",
          "service": "<http://secretmanager.googleapis.com|secretmanager.googleapis.com>"
        },
        "reason": "ACCOUNT_STATE_INVALID"
      }
    ]
I’m using pulumi ESC and OIDC
c
is your service account authorized for the secret manager?
c
Yep it has secret manager admin role
c
is the service account enabled / turned on? The error looks like the account state is bad... I'm guessing that's the service account state
you should be able to quickly tell by 
glcoud iam service-accounts list
and then 
gcloud iam service-accounts describe <some-service-account>
c
The service account should be good because it's used for all deployments through pulumi and every other stacks work fine
c
hmm, not sure then. I'm also fairly new to glcoud... I would try this next: • impersonate the service account • try performing the actions with gcloud (impersonating the service account) Hopefully that investigation leads you somewhere
c
When using Pulumi esc it automatically impersonates the service account. And the same happens
This is happening both in deployment and and cli using ESC
c
and it's just a single gcp project or stack? How do you have your stacks / projects set up?
c
For each project there are two stacks. Dev and prod. For this project this is currently happening only for dev stack because I don't want to test it on prod
c
and it's the same account across all projects?
c
Both dev and prod use different gcp account but they have same configuration because they are also managed through pulumi
So both projects are essentially mirror copies
@colossal-tailor-72573 I think i’ve found the issue. It’s not just secret manager operations but any operation at all. The email in the error is pulumi@ride-app-dev-2.iam.gserviceaccount.com but it should be pulumi@<project-number>.iam.gserviceaccount.com. I’m not impoersonatinve the svc acc on cli and ESC is configured correctly to use the correct email. idk why it’s happening
c
keep digging 🙂
c
@colossal-tailor-72573 after a lot of digging, still can’t figure out why
c
😞
c
Looks like somehow email Is being hard linked to the stack. Even if for dev stack I change the environment from app-dev to app-prod or for prod stack do the opposite it still sticks to that email. @colossal-tailor-72573 by any chance could it be a bug in the internal representation of pulumi state?
It is trying to use a deleted service account instead of the one defined in ESC. @plain-diamond-92898 can you help out here?
p
Checking with the team. (Apologies for the delay in response. Was out of office last few days)
@colossal-quill-8119 - Were you able to figure it out?
c
nope
it still has the email hard linked
p
Okay. To confirm, GCP OIDC is working for you, and Pull secrets from the GCP secret manager is working? You are facing an issue when you are trying to write a secret to GCP using the credential you obtained via OIDC?
c
OIDC with ESC is working fine for EVERY OTHER stack other than this
No operation in this stack works. Not only secrets
p
Pulumi up/refresh doesn't work either, correct? can you share your ESC definition, pulumi config file for a stack that's working, and for this stack that doesn't work. If you could also capture a video that'd be really helpful!
c
Sure. Give me half an hour and I'll send you a loom on your dm?
p
That works! Thank you.
c
sent
4 Views
Open in Slack
PreviousNext
Powered by