https://pulumi.com logo
Join Slack
Powered by
Also, what's a good way to do the <webhook-create-...
# kubernetes
w
Also, what's a good way to do the webhook-create-signed-cert.sh dance to automate creating a secret for the 
vpc-admission-webhook
?
@gorgeous-egg-16927 sorry to ping you but hoping you can quickly give me a direction and confirm there's no "easy" way to do this that I might have missed
For example, trying to cobble this together with pulumi stuff in dotnet/c#:
Copy code
// windows vpc admission webhook; <https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html>
            var windowsKey = new PrivateKey($"{prefix}-vpc-aw-key",
                new PrivateKeyArgs { Algorithm = "ECDSA" });

            var windowsCr = new CertRequest($"{prefix}-vpc-aw-cr",
                new CertRequestArgs
                {
                    KeyAlgorithm = "ECDSA",
                    PrivateKeyPem = windowsKey.PrivateKeyPem,
                    Subjects = { new CertRequestSubjectArgs { CommonName = "vpc-admission-webhook.kube-system.svc" } },
                    DnsNames =
                    {
                        "vpc-admission-webhook",
                        "vpc-admission-webhook.kube-system",
                        "vpc-admission-webhook.kube-system.svc"
                    }
                });

            var windowsCsr = new CertificateSigningRequest($"{prefix}-vpc-aw-csr",
                new CertificateSigningRequestArgs
                {
                    Metadata = new ObjectMetaArgs { Name = "vpc-admission-webhook" },
                    Spec = new CertificateSigningRequestSpecArgs
                    {
                        SignerName = "",
                        Request = windowsCr.CertRequestPem,
                        Groups = { "system:authenticated" },
                        Usages = { "digital signature", "key encipherment", "server auth" }
                    }
                },
                new CustomResourceOptions { Provider = kubeProvider });

            //TODO approve cert and wait for it

            var windowsSecret = new Secret($"{prefix}-vpc-aw-sec",
                new SecretArgs
                {
                    Metadata = new ObjectMetaArgs
                    {
                        Namespace = "kube-system",
                        Name = "vpc-admission-webhook-certs"
                    },
                    Data =
                    {
                        { "key.pem", windowsKey.PrivateKeyPem },
                        { "cert.pem", windowsCsr.Status.Apply(status => status.Certificate) }
                    }
                },
                new CustomResourceOptions { Provider = kubeProvider });
I'm not sure what to specify for the 
SignerName
in this case and not sure if I can somehow make the csr auto approve
Also, terraform have a similar kubernetes_certificate_signing_request with an 
auto_approve
argument but this is not available to me via pulumi AFAIK and I'm not sure what it's doing exactly.
g
Sorry, I’m not familiar with that particular API. In general, I’d suggest dynamic providers for this type of thing, but just found out yesterday that they aren’t currently supported for Go or .NET.
So, I’m not quite sure what to suggest. @billowy-army-68599 any ideas?
b
@worried-city-86458 it's been a while since I worked with this, but I believe the 
SignerName
can be anything you want, you just need to update the 
status
to be 
Approved
https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#approval-rejection-api-client
if you set the 
signerName
of 
<http://kubernetes.io/kube-apiserver-client-kubelet|kubernetes.io/kube-apiserver-client-kubelet>
i believe it'll autoapprove
fwiw, this is the weirdest of the k8s apis, and that's saying something...
w
@billowy-army-68599 yeah I noticed 
<http://kubernetes.io/kube-apiserver-client-kubelet|kubernetes.io/kube-apiserver-client-kubelet>
would auto approve, but wasn't sure what could use it, plus it says permitted key usages 
client auth
and this is for 
server auth
... unless I'm misunderstanding something? I'm not sure it's available but I'll try to find terraform source for their api above for some clues.
I found the kubernetes terraform provider source with auto approve: https://github.com/hashicorp/terraform-provider-kubernetes/blob/master/kubernetes/resource_kubernetes_certificate_signing_request.go#L114-L134 This is definitely not available via pulumi, right?
g
No, we don’t currently have any special logic for that resource. It’s just the CRUD operations.
w
With the following changes I've hacked together an auto approve in dotnet/c#:
Copy code
// windows vpc admission webhook; <https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html>
var windowsKey = new PrivateKey($"{prefix}-vpc-aw-key",
    new PrivateKeyArgs { Algorithm = "RSA" });

var windowsCr = new CertRequest($"{prefix}-vpc-aw-cr",
    new CertRequestArgs
    {
        KeyAlgorithm = "RSA",
        PrivateKeyPem = windowsKey.PrivateKeyPem,
        Subjects = { new CertRequestSubjectArgs { CommonName = "vpc-admission-webhook.kube-system.svc" } },
        DnsNames =
        {
            "vpc-admission-webhook",
            "vpc-admission-webhook.kube-system",
            "vpc-admission-webhook.kube-system.svc"
        }
    });

var windowsCsr = new CertificateSigningRequest($"{prefix}-vpc-aw-csr",
    new CertificateSigningRequestArgs
    {
        Metadata = new ObjectMetaArgs { Name = "vpc-admission-webhook" },
        Spec = new CertificateSigningRequestSpecArgs
        {
            Request = windowsCr.CertRequestPem.Apply(pem => Convert.ToBase64String(Encoding.ASCII.GetBytes(pem))),
            Groups = { "system:authenticated" },
            Usages = { "digital signature", "key encipherment", "server auth" }
        }
    },
    new CustomResourceOptions { Provider = kubeProvider });

var windowsSecret = new Secret($"{prefix}-vpc-aw-sec",
    new SecretArgs
    {
        Metadata = new ObjectMetaArgs
        {
            Namespace = "kube-system",
            Name = "vpc-admission-webhook-certs-2"
        },
        Data =
        {
            { "key.pem", windowsKey.PrivateKeyPem.Apply(pem => Convert.ToBase64String(Encoding.ASCII.GetBytes(pem))) },
            { "cert.pem", windowsCsr.AutoApprove(KubeConfig) }
        }
    },
    new CustomResourceOptions { Provider = kubeProvider });
Copy code
public static class CertificateSigningRequestExtensions
{
    public static Output<string> AutoApprove(this CertificateSigningRequest csr, Output<string> kubeConfig) =>
        Output.Tuple(csr.Metadata, kubeConfig)
            .Apply(async ((ObjectMeta Metadata, string KubeConfig) tuple) =>
            {
                if (Deployment.Instance.IsDryRun)
                {
                    return tuple.Metadata.Name;
                }

                var k8SConfig = Yaml.LoadFromString<K8SConfiguration>(tuple.KubeConfig);
                var clientConfig = KubernetesClientConfiguration.BuildConfigFromConfigObject(k8SConfig);
                var client = new Kubernetes(clientConfig);

                var issuedCert = Array.Empty<byte>();
                var issuedEvent = new AsyncManualResetEvent();
                await client.WatchCertificateSigningRequestAsync(tuple.Metadata.Name,
                    onEvent: (type, watchedCsr) =>
                    {
                        if (watchedCsr.Status.Conditions?.Any(condition => condition.Type == "Approved") == true &&
                            watchedCsr.Status.Certificate?.Length > 0)
                        {
                            issuedCert = watchedCsr.Status.Certificate;
                            issuedEvent.Set();
                        }
                    });

                var pendingCsr = await client.ReadCertificateSigningRequestAsync(tuple.Metadata.Name);
                pendingCsr.Status.Conditions ??= new List<V1beta1CertificateSigningRequestCondition>();
                if (pendingCsr.Status.Conditions.All(condition => condition.Type != "Approved"))
                {
                    Log.Debug("Approving certificate signing request...");
                    var approval = new V1beta1CertificateSigningRequestCondition
                    {
                        Type = "Approved",
                        Reason = "PulumiAutoApprove",
                        Message = "This CSR was approved by Pulumi auto approve."
                    };
                    pendingCsr.Status.Conditions.Add(approval);
                    await client.ReplaceCertificateSigningRequestApprovalAsync(pendingCsr, tuple.Metadata.Name);
                    Log.Debug("Approved certificate signing request");
                }

                Log.Debug("Waiting for certificate to be issued...");
                await Task.WhenAny(issuedEvent.WaitAsync(), Task.Delay(TimeSpan.FromMinutes(1)));
                if (!issuedEvent.IsSet)
                {
                    throw new TimeoutException("Timed out waiting for certificate to be issued");
                }
                Log.Debug("Waited for certificate to be issued");

                return Convert.ToBase64String(issuedCert);
            });
}
I resorted to using https://github.com/kubernetes-client/csharp to interact with the api
@gorgeous-egg-16927 @billowy-army-68599 now I've got the problem of how to actually do it properly; i.e. make it behave idempotently ... which I suspect is what dynamic resources would help me with?
CSRs are auto deleted by k8s after ~1h. The tls key and tls CR are only used to feed the k8s CSR. It's like this should just be a k8s secret resource with everything else being an implementation detail of that resource. Not sure how to proceed from here...
b
So you need something to accept the CSR right? You might have to write an operator that does it
w
The code above auto approves the CSR
The problem now is I'm not sure how to bundle this so that I avoid subsequent runs blowing up with:
Copy code
Type                                                                     Name                  Status         Info
     pulumi:pulumi:Stack                                                      aws-alpha             **failed**     1 error
 ~   └─ kubernetes:<http://admissionregistration.k8s.io:MutatingWebhookConfiguration|admissionregistration.k8s.io:MutatingWebhookConfiguration>  alpha-eks-vpc-aw-cfg  updated        [diff: ~apiVersion]

Diagnostics:
  pulumi:pulumi:Stack (aws-alpha):
    error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Aws\bin\Debug\Aws.dll' failed with an unhandled exception:
    Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'NotFound'
       at k8s.Kubernetes.ReadCertificateSigningRequestWithHttpMessagesAsync(String name, Nullable`1 exact, Nullable`1 export, String pretty, Dictionary`2 customHeaders, CancellationToken cancellationToken)
       at k8s.KubernetesExtensions.ReadCertificateSigningRequestAsync(IKubernetes operations, String name, Nullable`1 exact, Nullable`1 export, String pretty, CancellationToken cancellationToken)
       at Pharos.Gemini.CertificateSigningRequestExtensions.<>c.<<AutoApprove>b__0_0>d.MoveNext() in D:\Devel\Mps\devops-gemini-pulumi\Aws\Extensions\CertificateSigningRequestExtensions.cs:line 44
...
Since the CSR is transient (deleted after ~1h), regardless of approval.
Comparing to the terraform code that helped me get there, the lifetime of the k8s CSR is scoped to resourceKubernetesCertificateSigningRequestCreate i.e. it creates it and deletes it (when the function exits). There are also resourceKubernetesCertificateSigningRequestRead and resourceKubernetesCertificateSigningRequestDelete no-ops. Excuse my ignorance but is this what a pulumi dynamic provider would model? (in which case I'm screwed until there's dotnet/c# support)
6 Views
Open in Slack
PreviousNext
Powered by