https://pulumi.com logo
Powered by
Title
s

stocky-sundown-45608

01/16/2023, 4:10 PM
const issuer = new k8s.apiextensions.CustomResource(issuerName, {
  apiVersion: "<http://cert-manager.io/v1|cert-manager.io/v1>",
  kind: "Issuer",
  metadata: {
      namespace: appNs.metadata.name,
      name: issuerName,
  },
  spec: {
      acme: {
        server: "<https://acme-v02.api.letsencrypt.org/directory>",
        email: "",
        privateKeySecretRef: {
          name: issuerSecret.metadata.name,
        },
        solvers: [{
          http01: {
              ingress: {
                  name: "api-ingress-dev"
              },
          },
        }],
      },
  },
b

billowy-army-68599

01/16/2023, 4:12 PM
what error are you getting?
s

stocky-sundown-45608

01/16/2023, 4:14 PM
Hey, getting issuer fails to create (I will pull debug log) but once I remove solver it works. I also looked at your pulumi cert manager code repo where it was done using CRDs so I am wondering if anything is wrong using @pulumi/kubernetes-cert-manager”
while I pull up the trace, I am thinking is it related to the dependency in GCP where the solver needs to be mapped to ingress but ingress cannot have tls setup without creating an issuer and adding the issuer name in annotations. To be more clear around the issue the following steps are what I am doing • Setup Cert manager • create an ingress without tls • create an issuer with solver pointing to the above ingress • Update the ingress with tls and issuer name, check for certificate ready Last step is not clear on execution and could be causing this as I am not sure how to update the ingress and how the order matters after tls is created, also how to wait for certificate which takes long on gcp?
b

billowy-army-68599

01/16/2023, 4:31 PM
for the http solver to work, you need to be able to hit the url from the web and grab a value, so there could be a lot of things in the way • check the ingress url resolves correctly • check you can request the payload from the url
there isn’t enough information here to help you debug, once you have logs it’ll be clearer
s

stocky-sundown-45608

01/16/2023, 4:34 PM
sure I will add the logs, but is there a better way to update the ingress once the issuer is created. GCP issuer needs the ingress name unlike nginx,
there is cyclic dependency with the issuer -> ingress -> issuer
b

billowy-army-68599

01/16/2023, 4:39 PM
Without your current code it’s hard to say
s

stocky-sundown-45608

01/16/2023, 4:41 PM
https://gist.github.com/seeker815/1abb0f94adfbf342a39c3534b350b4c8
I have just added a sample, but the above + code explains - > ingress, issuer, ingress + tls . the last part is one I am not sure on how to get this immutable
b

billowy-army-68599

01/16/2023, 5:01 PM
Let me know when you have the error message, I don’t see anything immediately wrong with the code
s

stocky-sundown-45608

01/16/2023, 5:12 PM
Line number 8, 13-16 won’t run without the issuer being created,
line number 60 won’t run without ingress (w/ w/o tls)
I can add in logs, just need to re-create stack
b

billowy-army-68599

01/16/2023, 5:26 PM
hmm yeah i see the issue, Iv’e had this working before but it’s been a while
d

dry-keyboard-94795

01/16/2023, 5:57 PM
GCE Ingress is supposed to still configure the HTTP Forwarding Rules if the TLS is missing
What events are you getting:? 
kubectl --namespace <ns> describe ingress <ingressname>
s

stocky-sundown-45608

01/16/2023, 5:59 PM
that is correct, first time I can create ingress without tls. How do I update the ingress once the issuer is created? - Let’s say I repeat the ingress code, that won’t be immutable after the ingress with tls has been created
Also wait loop on certificate to be created and check on status ready, this takes a while in gcp/gke
to be clear - first run ingress without tls, create issuer, update ingress with tls, wait on certificate to be ready
last 2 steps is what I am looking to solve
d

dry-keyboard-94795

01/16/2023, 6:02 PM
what errors are you encountering?
s

stocky-sundown-45608

01/16/2023, 6:03 PM
I had an error with issuer but that got resolve after removing tls https://gist.github.com/seeker815/1abb0f94adfbf342a39c3534b350b4c8
so how do I now add tls post issuer creation
d

dry-keyboard-94795

01/16/2023, 6:04 PM
yes, what error?
s

stocky-sundown-45608

01/16/2023, 6:06 PM
I am more looking at pattern, in the above gist if I remove line number 8, 13-16 the ingress is created, and then issuer gets created. How do I then apply back tls and issuer name, I am looking at either editing ingress resource or recreating it?
d

dry-keyboard-94795

01/16/2023, 6:09 PM
Untested, but you could probably use an IngressPatch that depends on the Ingress + Issuer, and mark the 
tls
field as ignored on your Ingress resource. https://www.pulumi.com/registry/packages/kubernetes/api-docs/networking/v1/ingresspatch/
s

stocky-sundown-45608

01/16/2023, 6:11 PM
this will help, I can try it out and didn’t find it in the docs earlier.
Any suggestions for the wait on the certificate if it is around 10 minutes or more
b

billowy-army-68599

01/16/2023, 6:12 PM
Skipping the await will help a lot here
s

stocky-sundown-45608

01/16/2023, 6:14 PM
will try that out, thanks for your help to both 🙂
d

dry-keyboard-94795

01/16/2023, 6:20 PM
just to check, what is your 
issuerSecret
?
I've always let cert-manager create its own
s

stocky-sundown-45608

01/16/2023, 6:27 PM
according to cert-manager docs, for gcp we need to create an empty secret which gets populated with certificate later. For nginx ingress I have let cert-manager create the secret
d

dry-keyboard-94795

01/16/2023, 7:03 PM
out of my own curiosity, spinning this up on a cluster now..
wow, yeah. Just got back. I'm seeing 2 issues with this on my cluster • cert-manager really doesn't like you creating your own Secret like it suggests to do for GKE. This could be something on my cluster though, or something to do with SSA and it not wanting to take over after Pulumi has created the resource. • Reconciliation of the GCE Load Balancer is Slow, has regression in how it handles the 
tls
config, AND isn't registering the cert-manager backend to the URLMap consistently (it does create the backend though, and sometimes does the URLMap correctly)
at work, we use the DNS Solver for cert-manager, more for the fact that we only expose HTTPS. I guess this is just another reason to avoid the HTTP solvers, especially on GKE 🙂
s

stocky-sundown-45608

01/17/2023, 5:11 AM
I will replace the solver with dns but totally agree on GCE LB slow and cert-manager validation taking time. I will also try w/without secret and see how that goes. Appreciate your help trying this out
d

dry-keyboard-94795

01/17/2023, 11:24 AM
The slow parts I saw were mainly from GCE. cert-manager flew through everything once the webhook was live. GCE being slow is well documented, and gets much slower as you add new Ingresses or services attached to the Ingresses.
s

stocky-sundown-45608

01/22/2023, 6:47 AM
I am seeing issues with helm + node selector, would like your take on it if possible, https://pulumi-community.slack.com/archives/C84L4E3N1/p1674369763853879 your help is much appreciated.
#generalJoin Slack
Powered by