https://pulumi.com logo
Join Slack
Powered by
Hello here, Following my previous thread, I was ab...
# esc
f
Hello here, Following my previous thread, I was able to connect to gcp using oidc for a while (a few days), but I suddenly started to get those errors on all my existing resources (whether doing an 
up
,
refresh
or 
destroy
) :
Copy code
* Error when reading or editing ***REDACTED***: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See <https://developers.google.com/identity/sign-in/web/devconsole-project>.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "method": "cloud.dns.api.v1beta2.ManagedZonesService.Get",
      "service": "dns.googleapis.com"
    },
    "reason": "ACCESS_TOKEN_EXPIRED"
  }
]

More details:
Reason: authError, Message: Invalid Credentials
Nothing changed since I've setup the environment and everything was working, it just suddenly stops. The 
pulumi env open
shows that the token is refreshed each time with an expiration date in the future (10min)
Copy code
{
  "environmentVariables": {
    "CLOUDSDK_AUTH_ACCESS_TOKEN": "ya29.c.c0AY_***REDACTED***_VslI2r_t",
    "GOOGLE_PROJECT": ***REDACTED***
  },
  "gcp": {
    "login": {
      "accessToken": "ya29.c.c0AY_***REDACTED***_VslI2r_t",
      "expiry": "2024-04-24T10:00:20Z", // date is Wed Apr 24 09:50:22 UTC 2024
      "project": ***REDACTED***,
      "tokenType": "Bearer"
    }
  },
  "pulumiConfig": {
    "gcp:accessToken": "ya29.c.c0AY_***REDACTED***_VslI2r_t"
  }
}
I can also see the token properly refreshed when doing a 
pulumi config --show-secrets
Copy code
KEY              VALUE
gcp:accessToken  ya29.c.c0AY_***REDACTED***MRFW1g
gcp:project      ***REDACTED***
gcp:region       ***REDACTED***

ENVIRONMENT VARIABLE        VALUE
CLOUDSDK_AUTH_ACCESS_TOKEN  ya29.c.c0AY_***REDACTED***MRFW1g
GOOGLE_PROJECT              ***REDACTED***
some logs with 
-v=9
flags shows the token is set into the provider :
Copy code
I0424 09:41:45.593206    7914 rpc.go:77] Marshaling property for RPC[Provider[gcp, 0x400228d8c0].Configure().inputs]: accessToken={&{{ya29.c.c0AY_***REDACTED***syJc7XsaXMx}}}
I0424 09:41:45.593215    7914 rpc.go:77] Marshaling property for RPC[Provider[gcp, 0x400228d8c0].Configure().inputs]: ***REDACTED***={***REDACTED***}
I0424 09:41:45.593219    7914 rpc.go:77] Marshaling property for RPC[Provider[gcp, 0x400228d8c0].Configure().inputs]: value={ya29.c.c0AY_***REDACTED***syJc7XsaXMx}
I0424 09:41:45.593225    7914 rpc.go:77] Marshaling property for RPC[Provider[gcp, 0x400228d8c0].Configure().inputs]: project={***REDACTED***}
I0424 09:41:45.593227    7914 rpc.go:77] Marshaling property for RPC[Provider[gcp, 0x400228d8c0].Configure().inputs]: region={***REDACTED***}
I0424 09:41:45.593229    7914 rpc.go:77] Marshaling property for RPC[Provider[gcp, 0x400228d8c0].Configure().inputs]: version={7.18.0}
Early on in the logs I also noticed that one, not sure if that's related/relevant
Copy code
I0424 09:41:46.449346    7914 eventsink.go:59] [INFO] error retrieving userinfo for your provider credentials. have you enabled the '<https://www.googleapis.com/auth/userinfo.email>' scope? error: error retrieving userinfo for your provider credentials. have you enabled the '<https://www.googleapis.com/auth/userinfo.email>' scope? error: googleapi: got HTTP response code 401 with body: {
I0424 09:41:46.449541    7914 eventsink.go:62] eventSink::Debug(<{%reset%}>[INFO] error retrieving userinfo for your provider credentials. have you enabled the '<https://www.googleapis.com/auth/userinfo.email>' scope? error: error retrieving userinfo for your provider credentials. have you enabled the '<https://www.googleapis.com/auth/userinfo.email>' scope? error: googleapi: got HTTP response code 401 with body: {<{%reset%}>)
I0424 09:41:46.450457    7914 eventsink.go:59]   "error": "invalid_request",
I0424 09:41:46.450501    7914 eventsink.go:62] eventSink::Debug(<{%reset%}>  "error": "invalid_request",<{%reset%}>)
I0424 09:41:46.451200    7914 eventsink.go:59]   "error_description": "Invalid Credentials"
I0424 09:41:46.451304    7914 eventsink.go:62] eventSink::Debug(<{%reset%}>  "error_description": "Invalid Credentials"<{%reset%}>)
I0424 09:41:46.452081    7914 eventsink.go:59] }
I0424 09:41:46.452139    7914 eventsink.go:62] eventSink::Debug(<{%reset%}>}<{%reset%}>)
It just suddenly stopped working, I didn't change anything related to the oidc configuration nor in the pulumi env or stack config, I have no clue why nor how to fix this. I've ensured I'm not logged in with 
gcloud
locally in case that could disrupt the provider using the config/env settings, but that's not it. Any help to point me in any direction would be great, thanks 🙏 EDIT: Ok so after having tried to call the googleapis manually with 
curl
and every token I get by 
pulumi env
, 
pulumi config
and in the 
pulumi up
logs, it turns out the token 
pulumi up
retrieve is never refreshed, thus expired (at least not in the last hour I'm investigating) while the ones in the env and stack config change each time I call them (and they obviously authenticate properly) It seems there could be a cache or something fishy here. Should I open an issue ? ping @red-match-15116 since you were helping me with this last time Seems highly related to this thread https://pulumi-community.slack.com/archives/C0602S4P4T1/p1706617892755959 from @adventurous-butcher-54166
So, here's how I got out of this situation : • Created a new "manual" gcp provider (args had to be set from the config)
Copy code
const provider = new gcp.Provider('gcp-oidc-provider', {
  accessToken: gcpAccessToken,
  project: gcpProject,
  region: gcpRegion,
});
• 
pulumi up
that was able to create it but failed on every other resources (not able to explicitly update the provider in the 
CustomResourceOptions
apparently, while that's internal config, it does not need to do anything against the google apis. but still try to) • Had to export the state, replace manually the provider in each resource from the default one to the new custom one, then import the state back • 
pulumi up
finally works, I can see a 
[diff: ~accessToken]
on the gcp provider refreshing the token I'm really not happy with the current situation, now I have to explicitly set the provider in each and every resource to ensure the token is properly refreshed and to avoid getting stuck like this again. Is there another way to have a default provider actually refreshing tokens (like it should do) ? Why is that not explained in the docs ?
r
Hey @faint-motherboard-95438 this is a known issue. This comment should help you get around it: https://github.com/pulumi/esc/issues/199#issuecomment-2075198387 We're looking into fixing the underlying issue
f
Hey @red-match-15116 Thanks for getting back to me. I'm trying to get my head around it to avoid encountering it again, I've lost all my morning trying to get out of a stale state. I see this is known/opened for a long time now, while this is a very serious one, because you get stuck for good here. I had to try a surgical modification of the state that luckily worked, but I'm sure not everyone would have though about doing that. I mean at this point the doc is dangerous to follow and anyone trying to implement this will most likely encounter it. In the issue you're saying that's related to `pulumi refresh`/`pulumi destroy` not refreshing the creds and using the ones in the state, but I was doing 
pulumi up
only when I hit the problem, even doing it several time did not refresh the token. (EDIT: I forgot I did do a destroy which triggered the problem in the first place, I explain that later down this thread) I'm not sure what you're suggesting in the issue. You are saying "do not store the credentials in the state", I for sure did not do that, at least on purpose, I have no idea how to do that. From the documentation my env looks like this :
Copy code
values:
  environmentVariables:
    CLOUDSDK_AUTH_ACCESS_TOKEN: '${gcp.login.accessToken}'
    GOOGLE_PROJECT: '${gcp.login.project}'
  gcp:
    login:
      fn::open::gcp-login:
        oidc:
          providerId: pulumi-oidc-provider
          serviceAccount: pulumi-oidc-sa@12345.iam.gserviceaccount.com
          workloadPoolId: pulumi-oidc-identity-pool
        project: 12345
  pulumiConfig:
    gcp:accessToken: ${gcp.login.accessToken}
    gcp:project: ${gcp.login.project}
You are suggesting to use only environment variables, that means I should remove the 
pulumiConfig
section in my env ? I was not aware the default provider was automatically configuring itself with those values, even less "statifying" them in the state. So if I understand you correctly, my options here are : • default provider that set itself based on environment variables only, nothing related to the token should be put in the stack config by either an environment or a config set • a custom provider like I did earlier to "force" the token to refresh, using the stack configuration, but with the downside to explicitly use it in every resource Is that it ?
Well... I just tried as suggested in the issue you linked to use only the environment variables only (not sure which one depending on the documentation or issue) like so :
Copy code
values:
  environmentVariables:
    CLOUDSDK_AUTH_ACCESS_TOKEN: ${gcp.login.accessToken}
    GOOGLE_OAUTH_ACCESS_TOKEN: ${gcp.login.accessToken}
    GOOGLE_PROJECT: ${gcp.login.project}
  gcp:
    login:
      fn::open::gcp-login:
        oidc:
          providerId: pulumi-oidc-provider
          serviceAccount: pulumi-oidc-sa@12345.iam.gserviceaccount.com
          workloadPoolId: pulumi-oidc-identity-pool
        project: 12345
That does not work :
Copy code
error: Missing required configuration variable 'gcp:accessToken'
        please set a value using the command `pulumi config set gcp:accessToken <value>`
r
What version of the gcp provider are you using?
f
7.18.0
I see there's the 
7.19.0
but I don't see anything in the changelog related to that problem. I'll update it just in case.
r
hmm yeah this functionality was added in 7.11.0 so I'm not really sure what's going on
f
(pulumi CLI and package 
3.113.0
)
r
I have reached the extent of my knowledge of this issue, sorry @faint-motherboard-95438 - I've reached out internally to have someone help you out
f
No problem, thanks @red-match-15116 to have taken the time with me 🙏 In the meantime I'll try a few things on my own and let you guys know what I find out (do you prefer here while I investigate or in the issue you linked earlier ?)
@red-match-15116 ok I think I have the reproduction steps (that does not explain why the environment variables are ignored, I let that one for you to figure out 😉 ) • do a 
pulumi destroy
and have the token expiring right in the middle (yeah that sucks) • the resources are now in a unhealthy state some deleted, some not, and dependencies are a mess • try to do a 
pulumi up
to update the token, but since the resources are messed up by the previous aborted 
destroy
it actually fails and apparently does not update the token in the state because of that • aaaand... that's when you panic
Also, when I created the custom provider like so :
Copy code
const provider = new gcp.Provider('gcp-oidc-provider', {
  accessToken: gcpAccessToken,
  project: 'project',
  region: gcpRegion,
});
the 
accessToken
is stored in cleartext in the state instead of encrypted like in the default provider
r
the 
accessToken
is stored in cleartext in the state instead of encrypted like in the default provider
Y'know this is weird because this has also been fixed in recent versions of the provider. So something makes me think that somehow your provider version is out of date.
f
Indeed this is weird.. I just checked the version in the 
node_modules
, this is the right one according to the 
package.json
also that's a new project, so I actually started on this version, not on a previous one.
r
What's the output of 
pulumi about
f
Copy code
CLI          
Version      3.113.0
Go Version   go1.22.2
Go Compiler  gc

Plugins
NAME    VERSION
nodejs  unknown

Host     
OS       alpine
Version  3.19.1
Arch     aarch64

This project is written in nodejs: executable='/usr/local/bin/node' version='v20.12.2'

Current Stack: ***REDACTED***

TYPE                             URN
pulumi:pulumi:Stack              urn:pulumi:dev::core::pulumi:pulumi:Stack::***REDACTED***
pulumi:providers:pulumi          urn:pulumi:dev::core::pulumi:providers:pulumi::default
pulumi:pulumi:StackReference     urn:pulumi:dev::core::pulumi:pulumi:StackReference::***REDACTED***
pulumi:providers:gcp             urn:pulumi:dev::core::pulumi:providers:gcp::default_7_18_0

[...***REDACTED***...]
(i'm using a devcontainer if you're wondering about the alpine OS)
r
I may suggest opening a bug in pulumi-gcp. It's possible there's been a regression?
f
Alright, I’ll do a follow up there. thanks !
@red-match-15116 the PR you referenced ( https://github.com/pulumi/pulumi-gcp/pull/1715 ) was actually reverted by https://github.com/pulumi/pulumi-gcp/pull/1814 (both supporting env vars, which explain the error I get, and the secret for accessToken, which explain why I see it in clearText). It wanted to fix https://github.com/pulumi/pulumi-gcp/issues/1759 which is actually exactly my issue. Not sure if I read it properly, but this revert is going against your own documentation about the environment variable, and (re)introduce a security issue with the 
accessToken
being stored in cleartext in the state.
Open in Slack
PreviousNext
Powered by