https://pulumi.com logo
Powered by
w

wide-easter-61599

01/18/2019, 9:53 PM
I'm trying to run the pulumi/kubernetes-the-prod-way stack but it can't create the network. https://github.com/pulumi/kubernetes-the-prod-way/tree/master/gcp/infrastructure I added rolebindings for roles/compute.networkAdmin and roles/compute.securityAdmin to the user that the identity stack produces for this stuff, but I get an error when it tries to create the network. See thread for full error.
Copy code
error: Plan apply failed: Error creating network: Post <https://www.googleapis.com/compute/v1/projects/fine-sublime-216421/global/networks?alt=json&prettyPrint=false>: oauth2: cannot fetch token: 400 Bad Request
    Response: {
      "error": "invalid_grant",
      "error_description": "Bad Request"
    }
s

stocky-spoon-28903

01/18/2019, 9:54 PM
cc @creamy-potato-29402
w

wide-easter-61599

01/18/2019, 9:57 PM
which would explain why the issue hasn't cropped up yet
c

chilly-photographer-60932

01/18/2019, 10:00 PM
GKE requires API permissions in GCP
w

wide-easter-61599

01/18/2019, 10:00 PM
but I got an error when creating the network
is there a specific role that should be in the identity stack that's missing with that change? That's my guess...
c

creamy-potato-29402

01/18/2019, 10:02 PM
have you used the login script in 
./scripts
?
you have to login with that service account.
w

wide-easter-61599

01/18/2019, 10:03 PM
I did
c

creamy-potato-29402

01/18/2019, 10:03 PM
ah
can you 
gcloud auth list
?
w

wide-easter-61599

01/18/2019, 10:03 PM
I think in the commit referenced, my user (created in the identity section) doesn't have the role required to create a new network
Copy code
*       <mailto:infra-ci@fine-sublime-216421.iam.gserviceaccount.com|infra-ci@fine-sublime-216421.iam.gserviceaccount.com>
c

creamy-potato-29402

01/18/2019, 10:04 PM
that is possible, this is still a bit of a WIP
yeah so that means it’s very probably a roles issue
w

wide-easter-61599

01/18/2019, 10:04 PM
I added roles/compute.networkAdmin
c

creamy-potato-29402

01/18/2019, 10:05 PM
and that did NOT work?
w

wide-easter-61599

01/18/2019, 10:05 PM
oh wait
the IAM user doesn't have the role
even though the identity 
pulumi up
worked and said it was adding it
c

creamy-potato-29402

01/18/2019, 10:06 PM
I’ve seen this too… But in my case, the permission mysteriously disappeared.
w

wide-easter-61599

01/18/2019, 10:06 PM
i'll try to 
up
identity again
if it works i'll send a PR with the new role
c

creamy-potato-29402

01/18/2019, 10:06 PM
did you add it from the console?
w

wide-easter-61599

01/18/2019, 10:07 PM
says it's unchanged, but I don't see the role on the user in gcp console
I added it in the identity index.ts alongside cloudsqladmin and clusteradmin
c

creamy-potato-29402

01/18/2019, 10:07 PM
can you try 
pulumi refresh
and see what happens?
w

wide-easter-61599

01/18/2019, 10:08 PM
it had updates, including the iam role
c

creamy-potato-29402

01/18/2019, 10:08 PM
Right, so that means the state in GCP is different than the state you know about.
You should proceed, and then run pulumi up
w

wide-easter-61599

01/18/2019, 10:08 PM
that added it
thanks
(btw up didn't have anything to do once refresh was done)
c

creamy-potato-29402

01/18/2019, 10:09 PM
mmm
weird.
w

wide-easter-61599

01/18/2019, 10:11 PM
still failed to create the network but I'm going to blame eventual consistency and try again in a little bit
c

creamy-potato-29402

01/18/2019, 10:11 PM
same error?
w

wide-easter-61599

01/18/2019, 10:11 PM
bah same error
Copy code
jadams ~/p/v/a/g/infrastructure *feature/pulumi λ pulumi up                                                                                                         9s 261ms
Previewing update (agencyrocket-infrastructure):

     Type                             Name                                        Plan
     pulumi:pulumi:Stack              infrastructure-agencyrocket-infrastructure
 >-  ├─ pulumi:pulumi:StackReference  agencyrocket                                read
 +   ├─ gcp:compute:Network           staging                                     create
 +   ├─ gcp:compute:Subnetwork        staging                                     create
 +   └─ gcp:container:Cluster         staging                                     create

Resources:
    + 3 to create
    2 unchanged

Do you want to perform this update? yes
Updating (agencyrocket-infrastructure):

     Type                             Name                                        Status                  Info
     pulumi:pulumi:Stack              infrastructure-agencyrocket-infrastructure
 >-  ├─ pulumi:pulumi:StackReference  agencyrocket                                read
 +   └─ gcp:compute:Network           staging                                     **creating failed**     1 error

Diagnostics:
  gcp:compute:Network (staging):
    error: Plan apply failed: Error creating network: Post <https://www.googleapis.com/compute/v1/projects/fine-sublime-216421/global/networks?alt=json&prettyPrint=false>: oauth2: cannot fetch token: 400 Bad Request
    Response: {
      "error": "invalid_grant",
      "error_description": "Bad Request"
    }

Resources:
    2 unchanged

Duration: 4s

Permalink: <https://app.pulumi.com/knewter/agencyrocket-infrastructure/updates/10>
error: update failed
c

creamy-potato-29402

01/18/2019, 10:12 PM
can you try adding the permissions in the console just to make sure we’re not doing something silly?
I bet if you try refresh again it will say the permission is missing
w

wide-easter-61599

01/18/2019, 10:13 PM
seems to have them
c

creamy-potato-29402

01/18/2019, 10:14 PM
hold on let me look at my notes really quick
can you do a refresh again just to make sure?
that there are no changes
thanks for your patience, btw
Because I had created a new gcloud directory, I didn’t have any application default credentials; this was confirmed by running command gcloud auth application-default print-access-token. Before deleting the gcloud directory, I had application default credentials set, but the token must have been invalidated causing the invalid_grant error from above.
w

wide-easter-61599

01/19/2019, 2:55 AM
@creamy-potato-29402 sorry, got pulled away...for a long time
c

creamy-potato-29402

01/19/2019, 2:56 AM
no problem!
lmk what you think
w

wide-easter-61599

01/19/2019, 2:56 AM
it looks like that's what it is 100%. Sigh, thank you.
c

creamy-potato-29402

01/19/2019, 2:56 AM
Not your fault
I do wish we had a principled way of saying when something is “not our fault” lol
w

wide-easter-61599

01/19/2019, 2:57 AM
I have a stack for identity and a stack for infrastructure, different directories, and the thing partially worked so I didn't anticipate this
I don't really understand whose fault it would be
c

creamy-potato-29402

01/19/2019, 2:58 AM
perhaps no one’s
w

wide-easter-61599

01/19/2019, 2:58 AM
welp I re-authed but get the same problem
oh that access token still won't print
c

creamy-potato-29402

01/19/2019, 2:58 AM
🤔
w

wide-easter-61599

01/19/2019, 2:58 AM
let me read your links in more detail 🙂
c

creamy-potato-29402

01/19/2019, 2:58 AM
when we figure it out it will be useful for future travelers, I think
w

wide-easter-61599

01/19/2019, 2:59 AM
aha
lol
Copy code
jadams ~/p/v/a/g/infrastructure *feature/pulumi λ gcloud auth application-default revoke                                                                           1 1s 60ms
You are about to revoke the credentials stored in:
[/Users/jadams/.config/gcloud/application_default_credentials.json]

Do you want to continue (Y/n)?

ERROR: gcloud crashed (TokenRevokeError): invalid_token

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics
turns out I also have lots of gcloud component updates so maybe that's related ¯\_(ツ)_/¯ i'll get there
c

creamy-potato-29402

01/19/2019, 3:01 AM
😬
eek.
w

wide-easter-61599

01/19/2019, 3:03 AM
seems like this fixed it:
Copy code
gcloud auth application-default login
which confirms that was causing it
why is it using the application default token? Is that a thing pulumi should change?
c

creamy-potato-29402

01/19/2019, 5:58 AM
@wide-easter-61599 hmm, what do you think the alternative is?
w

wide-easter-61599

01/19/2019, 6:09 PM
I'm not sure, I thought that it would be using the identity from the first stack for everything.
Powered by