Is it possible to wait for or otherwise query a `k...
# general
r
Is it possible to wait for or otherwise query a
k8s.apiextensions.CustomResource
or
k8s.yaml.ConfigFile
to succeed? I see there is an issue tracking to make it type safe. Pulumi applies the resource fine, but I need to then check to make sure it didn’t fail. I see pulumi query, but that doesn’t appear to work with pulumi up.
g
`CustomResource`s don’t have built in readiness logic, but the underlying resources of a
ConfigFile
often do (Deployments, Services, etc). https://github.com/pulumi/pulumi-kubernetes/issues/861 is still open, but in the meantime, you can use
dependsOn
with
getResource
as shown here: https://github.com/pulumi/pulumi-kubernetes/blob/0162c7f3ff223c70eb5f29d70cec77d8dbaf52a7/tests/integration/istio/step1/istio.ts#L43-L69
r
Thank you. I have attempted to use the
getResource
but it appears to be ignored.
Copy code
// Create TLS secret from sealed secret
const tlsCert = new k8s.yaml.ConfigFile('dev-local-tls', {
  file: '../01-identity/dex/sealed-dev-local-tls.yaml',
})


// SealedSecrets controller will decrypt the above and generate a
// dev-local-tls-ss-test secret in the identity namespace


// In this example this should fail as the sealed-secret will fail to decrypt...
const secret = tlsCert.getResource(
  'v1/Secret',
  'identity',
  'dev-local-tls-ss-test'
)


const next = new k8s.core.v1.Secret('should-wait', {}, { dependsOn: secret })
g
Is the Secret part of the YAML, or generated by a controller? If it’s generated by a controller, this is a case we don’t have a built-in answer for. You might have a add a polling loop or something to make sure the Secret exists before continuing. Perhaps it would make sense to add optional retries to the
get
operations to make this easier? @white-balloon-205
r
It is a Bitnami Sealed Secret https://github.com/bitnami-labs/sealed-secrets#sealed-secrets-for-kubernetes The SealedSecret itself has the appropriate events that should be queryable (i.e. failed to decrypt, unsealed, etc) the controller then creates a Secret. It doesn’t appear that CustomResources can be queried with the getResource method (or at least the TS types don’t allow for it). So the approach would be to either query the SealedSecret itself to see if it has an error, or somehow reference the generated secret created by the controller.
I ran into this because a later resource (an ingress) referenced a secret by its string name. The secret was never created on kubernetes because the bitnami sealedsecret controller rejected the sealedsecret. So the ingress was unhealthy but the pulumi program failed to detect that. I imagine the issue would be the same if the secret came from a jetstack cert-manager controller as well.
g
Thanks for the info. Would you mind opening an issue on that? I’ve got some ideas on how to make that type of thing easier with pulumi
r
What is confusing is that in the code example I posted above
Copy code
const secret = tlsCert.getResource()
seems to be a no-op. I know the resource doesn’t exist. So I would expect anything else that dependsOn it would fail.
g
I suspect what’s happening in that case is that the return value is
undefined
, and therefore doesn’t actually change the
dependsOn
.
Would need to dig into it further
r
Oh. That makes sense.
What would the issue be for specifically @gorgeous-egg-16927
g
The scenario you described with the sealed secrets. Just need more context on what you were expecting vs what actually happened
r
Will do
🙏 1
https://github.com/pulumi/pulumi-kubernetes/issues/912 Not as polished of an issue as I would like, but hopefully it helps.
Looking through the code, I see pkg/await/awaiters.go handles the readiness logic. And in await.go it is naturally assumed that if a custom resource is applied, it is done. Perhaps a means to register external awaiters would be a good extension point. Or I could look into contributing some of the popular CRDs. @gorgeous-egg-16927
g
That looks great; thanks for writing that up! We’re definitely looking for ways to improve the CRD experience on Pulumi. Most folks are currently writing custom logic in their programs to handle those cases, but I do like the idea of making the provider itself more extensible.