https://pulumi.com logo
Join Slack
Powered by
I'm trying to run the pulumi/kubernetes-the-prod-w...
# general
w
I'm trying to run the pulumi/kubernetes-the-prod-way stack but it can't create the network. https://github.com/pulumi/kubernetes-the-prod-way/tree/master/gcp/infrastructure I added rolebindings for roles/compute.networkAdmin and roles/compute.securityAdmin to the user that the identity stack produces for this stuff, but I get an error when it tries to create the network. See thread for full error.
Copy code
error: Plan apply failed: Error creating network: Post <https://www.googleapis.com/compute/v1/projects/fine-sublime-216421/global/networks?alt=json&prettyPrint=false>: oauth2: cannot fetch token: 400 Bad Request
    Response: {
      "error": "invalid_grant",
      "error_description": "Bad Request"
    }
s
cc @creamy-potato-29402
w
which would explain why the issue hasn't cropped up yet
c
GKE requires API permissions in GCP
w
but I got an error when creating the network
is there a specific role that should be in the identity stack that's missing with that change? That's my guess...
c
have you used the login script in 
./scripts
?
you have to login with that service account.
w
I did
c
ah
can you 
gcloud auth list
?
w
I think in the commit referenced, my user (created in the identity section) doesn't have the role required to create a new network
Copy code
*       <mailto:infra-ci@fine-sublime-216421.iam.gserviceaccount.com|infra-ci@fine-sublime-216421.iam.gserviceaccount.com>
c
that is possible, this is still a bit of a WIP
yeah so that means it’s very probably a roles issue
w
I added roles/compute.networkAdmin
c
and that did NOT work?
w
oh wait
the IAM user doesn't have the role
even though the identity 
pulumi up
worked and said it was adding it
c
I’ve seen this too… But in my case, the permission mysteriously disappeared.
w
i'll try to 
up
identity again
if it works i'll send a PR with the new role
c
did you add it from the console?
w
says it's unchanged, but I don't see the role on the user in gcp console
I added it in the identity index.ts alongside cloudsqladmin and clusteradmin
c
can you try 
pulumi refresh
and see what happens?
w
it had updates, including the iam role
c
Right, so that means the state in GCP is different than the state you know about.
You should proceed, and then run pulumi up
w
that added it
thanks
(btw up didn't have anything to do once refresh was done)
c
mmm
weird.
w
still failed to create the network but I'm going to blame eventual consistency and try again in a little bit
c
same error?
w
bah same error
Copy code
jadams ~/p/v/a/g/infrastructure *feature/pulumi λ pulumi up                                                                                                         9s 261ms
Previewing update (agencyrocket-infrastructure):

     Type                             Name                                        Plan
     pulumi:pulumi:Stack              infrastructure-agencyrocket-infrastructure
 >-  ├─ pulumi:pulumi:StackReference  agencyrocket                                read
 +   ├─ gcp:compute:Network           staging                                     create
 +   ├─ gcp:compute:Subnetwork        staging                                     create
 +   └─ gcp:container:Cluster         staging                                     create

Resources:
    + 3 to create
    2 unchanged

Do you want to perform this update? yes
Updating (agencyrocket-infrastructure):

     Type                             Name                                        Status                  Info
     pulumi:pulumi:Stack              infrastructure-agencyrocket-infrastructure
 >-  ├─ pulumi:pulumi:StackReference  agencyrocket                                read
 +   └─ gcp:compute:Network           staging                                     **creating failed**     1 error

Diagnostics:
  gcp:compute:Network (staging):
    error: Plan apply failed: Error creating network: Post <https://www.googleapis.com/compute/v1/projects/fine-sublime-216421/global/networks?alt=json&prettyPrint=false>: oauth2: cannot fetch token: 400 Bad Request
    Response: {
      "error": "invalid_grant",
      "error_description": "Bad Request"
    }

Resources:
    2 unchanged

Duration: 4s

Permalink: <https://app.pulumi.com/knewter/agencyrocket-infrastructure/updates/10>
error: update failed
c
can you try adding the permissions in the console just to make sure we’re not doing something silly?
I bet if you try refresh again it will say the permission is missing
w
seems to have them
c
hold on let me look at my notes really quick
can you do a refresh again just to make sure?
that there are no changes
thanks for your patience, btw
Because I had created a new gcloud directory, I didn’t have any application default credentials; this was confirmed by running command gcloud auth application-default print-access-token. Before deleting the gcloud directory, I had application default credentials set, but the token must have been invalidated causing the invalid_grant error from above.
w
@creamy-potato-29402 sorry, got pulled away...for a long time
c
no problem!
lmk what you think
w
it looks like that's what it is 100%. Sigh, thank you.
c
Not your fault
I do wish we had a principled way of saying when something is “not our fault” lol
w
I have a stack for identity and a stack for infrastructure, different directories, and the thing partially worked so I didn't anticipate this
I don't really understand whose fault it would be
c
perhaps no one’s
w
welp I re-authed but get the same problem
oh that access token still won't print
c
🤔
w
let me read your links in more detail 🙂
c
when we figure it out it will be useful for future travelers, I think
w
aha
lol
Copy code
jadams ~/p/v/a/g/infrastructure *feature/pulumi λ gcloud auth application-default revoke                                                                           1 1s 60ms
You are about to revoke the credentials stored in:
[/Users/jadams/.config/gcloud/application_default_credentials.json]

Do you want to continue (Y/n)?

ERROR: gcloud crashed (TokenRevokeError): invalid_token

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics
turns out I also have lots of gcloud component updates so maybe that's related ¯\_(ツ)_/¯ i'll get there
c
😬
eek.
w
seems like this fixed it:
Copy code
gcloud auth application-default login
which confirms that was causing it
why is it using the application default token? Is that a thing pulumi should change?
c
@wide-easter-61599 hmm, what do you think the alternative is?
w
I'm not sure, I thought that it would be using the identity from the first stack for everything.
2 Views
Open in Slack
PreviousNext
Powered by