https://pulumi.com logo
Join Slack
Powered by
I’m trying to setup cert manager and think I’m on ...
# general
i
I’m trying to setup cert manager and think I’m on my last hurdle. I’m using a restricted gcp service account for app deployments (from kube the prod way) and I’m bumping into:
Copy code
<http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> is forbidden: User "<mailto:ci-app@xxx.iam.gserviceaccount.com|ci-app@xxx.iam.gserviceaccount.com>" cannot create resource "clusterroles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" at the cluster scope
I assume I have to run the equivalent of:
Copy code
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user [USER_ACCOUNT]
But isn’t that the same as just granting gcp 
roles/container.clusterAdmin
? Here’s a gist of the current setup: https://gist.github.com/rosskevin/e80dabe6347fa34c179b3885e4f4a3a0 I’m a bit confused as to the right thing to do here and keep with the minimal permissions for the service account.
c
@important-leather-28796 why not just change your identity stack to have the permissions you need?
If that’s cluster admin?
i
I’m looking at that just not sure what I need
does gcp iam clusteAdmin ===
Copy code
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user [USER_ACCOUNT]
if so, the ktpw ciInf and ciApp identities aren’t very different
c
what does 
cluster-admin
role grant you in your cluster?
presumably it’s administrative permissions for your cluster? That could be the same
b
On GKE you're logging into the cluster with your Google Account email address, which by default doesn't have 
cluster-admin
capabilities.
i
in this case, I’m using the ktpw activated service account
which is ciApp in this case
c
you should separate out “kubernetes infrastructure” from app
I’d deploy clusterroles etc from the infra service account
i
That seems nice but impractical
c
Why?
i
These are needed as part of many charts including cert-manager and prometheus
c
Most of them let you source that stuff to already-created infrastructure.
Many, many large companies do it this way. It’s a pain, but that’s because charts are not designed for them.
i
c
If you are ok with granting your app god mode permissions, then the way you describe is ok
if you are not, you will have to split them out
i
seems like a violation of encapsulation.
so bad permissions, bad coding
bad all around
These clusterroles are part of certmanager, which is a reused component, used in a stack that is 2x removed from identity stack and 1x removed from our infra stack
c
So in the case of cert manager, I’d just provision that chart from your infra stack
that’s really important. Same with external-dns, prometheus, and so on.
for like elasticsearch or something? split.
i
I’ve created a 
resources
stack which is below our 
app
stack
so I guess the same idea
just need to use the infra service account
but I’ll need to add container deploy privilege to the infra sa
at least that would yield one limited app sa, instead of both being god mode
Ok, so even trying this with the inf sa, I’m seeing the same 
<http://clusterrolebindings.rbac.authorization.k8s.io|clusterrolebindings.rbac.authorization.k8s.io> is forbidden: User "<mailto:ci-infrastructure@xxx.iam.gserviceaccount.com|ci-infrastructure@xxx.iam.gserviceaccount.com>" cannot create resource "clusterrolebindings" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" at the cluster scope
. Updated gist https://gist.github.com/rosskevin/e80dabe6347fa34c179b3885e4f4a3a0
what am I missing?
c
what service account are you using, and does it have those permissions?
i
ci-infrastructure, bindings in the gist which is the gcp 
roles/container.clusterAdmin
and 
roles/container.developer
c
can you run 
pulumi refresh
?
i
c
I’m not sure how the GKE role mappings work
I think the command you’re showing earlier is actually creatinga role binding for your service account
not an IAM binding, but a kubernetes 
RoleBinding
object.
I don’t see that in there.
i
right, I think so too.
c
can you check if your user is allocated that 
RoleBinding
?
if not, then do that.
i
does that mean I 
new
a 
ClusterRoleBinding
in pulumi to mimic
Copy code
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user [sa email]
c
yes.
I think it does.
not an expert though
i
gcp console is hosed today, can’t see it but I know for sure it doesn’t have it unless granted
I’ll 
new
one and see where that goes, on the ci-inf sa
in the identity stack
c
yeah
i
I found the rough equivalent role binding in the istio integration test: https://github.com/pulumi/pulumi-kubernetes/blob/master/tests/integration/istio/step1/istio.ts working on adding that to the 
infrastructure
stack post-cluster creation
@creamy-potato-29402 I exported 
ciInf
service account from 
identity
. I want to import it and get the email. Do I need to parse the json from it if I don’t export the string email? e.g.
Copy code
stack.identity.getOutput('ciInf').apply(json => JSON.parse(json).email)
c
uhm
I forget what what happens when you export the service account.
I think it doesn’t do what you want? could be wrong.
i
I see a json in the output
c
is email in there?
if so you would have to parse it, yes.
i
yes
ok, so as getOutput we get what we see in the web console as a string - always
c
yeah
i
getting a json parse error is why I ask. is there a specific cli flag to see what it is pulling/breaking on parsing?
c
which error, now?
what does the string look like when you print it?
i
SyntaxError: Unexpected token o in JSON at position 1
export const foo = stack.identity.getOutput('ciInfrastructure')
yields
Copy code
Outputs:
    foo      : {
        accountId  : "ci-infrastructure"
        displayName: "CI infrastructure account"
        email      : "<mailto:ci-infrastructure@xxx.iam.gserviceaccount.com|ci-infrastructure@xxx.iam.gserviceaccount.com>"
        id         : "projects/advisorintake/serviceAccounts/ci-infrastructure@xxx.iam.gserviceaccount.com"
        name       : "projects/advisorintake/serviceAccounts/ci-infrastructure@xxx.iam.gserviceaccount.com"
        project    : "xxx"
        uniqueId   : "117964030702708620539"
        urn        : "urn:pulumi:development::xxx-identity::gcp:serviceAccount/account:Account::ciInfrastructure"
    }
c
Right but what does printing 
getOutput
show?
It looks like it’s not real JSON
based on the error
i
c
yeah
like literally what is in that thing
the error is basically saying it doesn’t parse, let’s find out why
i
oh
Copy code
foo is: OutputImpl {
      __pulumiOutput: true,
      isKnown: Promise { <pending> },
      resources: [Function],
      promise: [Function],
      toString: [Function],
      toJSON: [Function],
      apply: [Function],
      get: [Function] }
need to 
output
it perhaps
oh, it isn’t json, it is like a js string
c
I’m going to guess what’s happening here
i
keys aren’t quoted
c
we don’t support 
export
of 
CustomResource
. cc @microscopic-florist-22719 Instead you should just export the email.
i
ok
m
we do support custom resources as outputs, but we turn them into POJOs. You should be able to pull the email property out using an apply.
so given 
const foo = stack.identity.getOutput('ciInfrastructure')
, you can do 
foo.apply(foo => foo.email)
c
ah
i
yes, that works
thank you
c
lol my bad I assumed that we had just dumped the CR to string and passed that to the stack output.
i
^^ my assumption too
Open in Slack
PreviousNext
Powered by