https://pulumi.com logo
#general
Title
# general
i

important-leather-28796

03/11/2019, 6:33 PM
I’m trying to setup cert manager and think I’m on my last hurdle. I’m using a restricted gcp service account for app deployments (from kube the prod way) and I’m bumping into:
Copy code
<http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> is forbidden: User "<mailto:ci-app@xxx.iam.gserviceaccount.com|ci-app@xxx.iam.gserviceaccount.com>" cannot create resource "clusterroles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" at the cluster scope
I assume I have to run the equivalent of:
Copy code
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user [USER_ACCOUNT]
But isn’t that the same as just granting gcp
roles/container.clusterAdmin
? Here’s a gist of the current setup: https://gist.github.com/rosskevin/e80dabe6347fa34c179b3885e4f4a3a0 I’m a bit confused as to the right thing to do here and keep with the minimal permissions for the service account.
c

creamy-potato-29402

03/11/2019, 7:23 PM
@important-leather-28796 why not just change your identity stack to have the permissions you need?
If that’s cluster admin?
i

important-leather-28796

03/11/2019, 7:24 PM
I’m looking at that just not sure what I need
does gcp iam clusteAdmin ===
Copy code
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user [USER_ACCOUNT]
if so, the ktpw ciInf and ciApp identities aren’t very different
c

creamy-potato-29402

03/11/2019, 7:26 PM
what does
cluster-admin
role grant you in your cluster?
presumably it’s administrative permissions for your cluster? That could be the same
b

busy-pizza-73563

03/11/2019, 7:26 PM
On GKE you're logging into the cluster with your Google Account email address, which by default doesn't have
cluster-admin
capabilities.
i

important-leather-28796

03/11/2019, 7:27 PM
in this case, I’m using the ktpw activated service account
which is ciApp in this case
c

creamy-potato-29402

03/11/2019, 7:28 PM
you should separate out “kubernetes infrastructure” from app
I’d deploy clusterroles etc from the infra service account
i

important-leather-28796

03/11/2019, 7:28 PM
That seems nice but impractical
c

creamy-potato-29402

03/11/2019, 7:28 PM
Why?
i

important-leather-28796

03/11/2019, 7:29 PM
These are needed as part of many charts including cert-manager and prometheus
c

creamy-potato-29402

03/11/2019, 7:29 PM
Most of them let you source that stuff to already-created infrastructure.
Many, many large companies do it this way. It’s a pain, but that’s because charts are not designed for them.
i

important-leather-28796

03/11/2019, 7:29 PM
c

creamy-potato-29402

03/11/2019, 7:30 PM
If you are ok with granting your app god mode permissions, then the way you describe is ok
if you are not, you will have to split them out
i

important-leather-28796

03/11/2019, 7:31 PM
seems like a violation of encapsulation.
so bad permissions, bad coding
bad all around
These clusterroles are part of certmanager, which is a reused component, used in a stack that is 2x removed from identity stack and 1x removed from our infra stack
c

creamy-potato-29402

03/11/2019, 7:32 PM
So in the case of cert manager, I’d just provision that chart from your infra stack
that’s really important. Same with external-dns, prometheus, and so on.
for like elasticsearch or something? split.
i

important-leather-28796

03/11/2019, 7:33 PM
I’ve created a
resources
stack which is below our
app
stack
so I guess the same idea
just need to use the infra service account
but I’ll need to add container deploy privilege to the infra sa
at least that would yield one limited app sa, instead of both being god mode
Ok, so even trying this with the inf sa, I’m seeing the same
<http://clusterrolebindings.rbac.authorization.k8s.io|clusterrolebindings.rbac.authorization.k8s.io> is forbidden: User "<mailto:ci-infrastructure@xxx.iam.gserviceaccount.com|ci-infrastructure@xxx.iam.gserviceaccount.com>" cannot create resource "clusterrolebindings" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" at the cluster scope
. Updated gist https://gist.github.com/rosskevin/e80dabe6347fa34c179b3885e4f4a3a0
what am I missing?
c

creamy-potato-29402

03/11/2019, 7:47 PM
what service account are you using, and does it have those permissions?
i

important-leather-28796

03/11/2019, 7:48 PM
ci-infrastructure, bindings in the gist which is the gcp
roles/container.clusterAdmin
and
roles/container.developer
c

creamy-potato-29402

03/11/2019, 7:48 PM
can you run
pulumi refresh
?
i

important-leather-28796

03/11/2019, 7:49 PM
c

creamy-potato-29402

03/11/2019, 7:49 PM
I’m not sure how the GKE role mappings work
I think the command you’re showing earlier is actually creatinga role binding for your service account
not an IAM binding, but a kubernetes
RoleBinding
object.
I don’t see that in there.
i

important-leather-28796

03/11/2019, 7:50 PM
right, I think so too.
c

creamy-potato-29402

03/11/2019, 7:51 PM
can you check if your user is allocated that
RoleBinding
?
if not, then do that.
i

important-leather-28796

03/11/2019, 7:52 PM
does that mean I
new
a
ClusterRoleBinding
in pulumi to mimic
Copy code
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user [sa email]
c

creamy-potato-29402

03/11/2019, 7:52 PM
yes.
I think it does.
not an expert though
i

important-leather-28796

03/11/2019, 7:52 PM
gcp console is hosed today, can’t see it but I know for sure it doesn’t have it unless granted
I’ll
new
one and see where that goes, on the ci-inf sa
in the identity stack
c

creamy-potato-29402

03/11/2019, 7:53 PM
yeah
i

important-leather-28796

03/11/2019, 8:04 PM
I found the rough equivalent role binding in the istio integration test: https://github.com/pulumi/pulumi-kubernetes/blob/master/tests/integration/istio/step1/istio.ts working on adding that to the
infrastructure
stack post-cluster creation
@creamy-potato-29402 I exported
ciInf
service account from
identity
. I want to import it and get the email. Do I need to parse the json from it if I don’t export the string email? e.g.
Copy code
stack.identity.getOutput('ciInf').apply(json => JSON.parse(json).email)
c

creamy-potato-29402

03/11/2019, 8:40 PM
uhm
I forget what what happens when you export the service account.
I think it doesn’t do what you want? could be wrong.
i

important-leather-28796

03/11/2019, 8:40 PM
I see a json in the output
c

creamy-potato-29402

03/11/2019, 8:40 PM
is email in there?
if so you would have to parse it, yes.
i

important-leather-28796

03/11/2019, 8:41 PM
yes
ok, so as getOutput we get what we see in the web console as a string - always
c

creamy-potato-29402

03/11/2019, 8:42 PM
yeah
i

important-leather-28796

03/11/2019, 8:44 PM
getting a json parse error is why I ask. is there a specific cli flag to see what it is pulling/breaking on parsing?
c

creamy-potato-29402

03/11/2019, 8:44 PM
which error, now?
what does the string look like when you print it?
i

important-leather-28796

03/11/2019, 8:44 PM
SyntaxError: Unexpected token o in JSON at position 1
export const foo = stack.identity.getOutput('ciInfrastructure')
yields
Copy code
Outputs:
    foo      : {
        accountId  : "ci-infrastructure"
        displayName: "CI infrastructure account"
        email      : "<mailto:ci-infrastructure@xxx.iam.gserviceaccount.com|ci-infrastructure@xxx.iam.gserviceaccount.com>"
        id         : "projects/advisorintake/serviceAccounts/ci-infrastructure@xxx.iam.gserviceaccount.com"
        name       : "projects/advisorintake/serviceAccounts/ci-infrastructure@xxx.iam.gserviceaccount.com"
        project    : "xxx"
        uniqueId   : "117964030702708620539"
        urn        : "urn:pulumi:development::xxx-identity::gcp:serviceAccount/account:Account::ciInfrastructure"
    }
c

creamy-potato-29402

03/11/2019, 8:47 PM
Right but what does printing
getOutput
show?
It looks like it’s not real JSON
based on the error
i

important-leather-28796

03/11/2019, 8:48 PM
c

creamy-potato-29402

03/11/2019, 8:48 PM
yeah
like literally what is in that thing
the error is basically saying it doesn’t parse, let’s find out why
i

important-leather-28796

03/11/2019, 8:48 PM
oh
Copy code
foo is: OutputImpl {
      __pulumiOutput: true,
      isKnown: Promise { <pending> },
      resources: [Function],
      promise: [Function],
      toString: [Function],
      toJSON: [Function],
      apply: [Function],
      get: [Function] }
need to
output
it perhaps
oh, it isn’t json, it is like a js string
c

creamy-potato-29402

03/11/2019, 8:54 PM
I’m going to guess what’s happening here
i

important-leather-28796

03/11/2019, 8:54 PM
keys aren’t quoted
c

creamy-potato-29402

03/11/2019, 8:54 PM
we don’t support
export
of
CustomResource
. cc @microscopic-florist-22719 Instead you should just export the email.
i

important-leather-28796

03/11/2019, 8:54 PM
ok
m

microscopic-florist-22719

03/11/2019, 8:56 PM
we do support custom resources as outputs, but we turn them into POJOs. You should be able to pull the email property out using an apply.
so given
const foo = stack.identity.getOutput('ciInfrastructure')
, you can do
foo.apply(foo => foo.email)
c

creamy-potato-29402

03/11/2019, 8:58 PM
ah
i

important-leather-28796

03/11/2019, 8:58 PM
yes, that works
thank you
c

creamy-potato-29402

03/11/2019, 8:58 PM
lol my bad I assumed that we had just dumped the CR to string and passed that to the stack output.
i

important-leather-28796

03/11/2019, 8:58 PM
^^ my assumption too