https://pulumi.com logo
#general
Title
# general
f

full-eve-52536

09/07/2023, 3:52 PM
Does anyone know if Pulumi resources have a built-in
retry
mechanism? We sometimes have resource creation fail due to environmental issues. This is usually resolved when re-running
Pulumi Up
, but we would rather just retry a single resource operation, rather than the whole stack.
s

stocky-restaurant-98004

09/07/2023, 5:18 PM
Curious why you would only want to retry a single resource rather than have Pulumi continue to create the rest of the dependent resources in the stack.
f

full-eve-52536

09/07/2023, 5:19 PM
In my case, the
Release
resource of the kubernetes provider sometimes fails because the CRDs are not ready yet on the cluster, so ideally we would just have that
Release
resource try to install the helm chart again.
d

dry-keyboard-94795

09/07/2023, 5:23 PM
Also interested in this, for a similar reason where the Release resource is flakey. I just have a retry in place in the CI workflow to rerun the stack
Even if it's just a retry for helm release; higher chance of things not working first time. I get that helm charts are ubiquitous, but I've never really known them to be stable.
s

stocky-restaurant-98004

09/07/2023, 8:37 PM
I'm not super deep with the K8s provider, but can you add
depends_on=[my_crd.ready]
? I know that some resource in that provider does have a
ready
output.
@full-eve-52536 In this scenario, are there other resources in your stack would be retried that should not be? I'm asking b/c nothing that succeeded should be retried in a normal Pulumi program. Trying to figure out if there's a potential bug.
f

full-eve-52536

09/07/2023, 8:42 PM
In this scenario, are there other resources in your stack would be retried that should not be? I'm asking b/c nothing that succeeded should be retried in a normal Pulumi program. Trying to figure out if there's a potential bug.
No everything else seems to be working as expected. I think this "retry" feature may also be handy with other resources, too. For example,
Command.Local
may be running a curl command that fails because of an intermittent network issue. In that case, it would be nice to tell Pulumi to only retry that resource so I don't have to run the entire stack deployment.
s

stocky-restaurant-98004

09/07/2023, 8:43 PM
Ok, so use case is "make this resource more fault tolerant so I don't have to re-run pulumi up" (presumably because the preview takes longer than you want). Is that right?
BTW, you can use
command.local
to run
sleep
, which is how I solved for stuff not being ready in another codebase. I wouldn't call it ironclad, but it worked pretty reliably. Then you can make your Helm chart resource depend on the command.
f

full-eve-52536

09/07/2023, 8:49 PM
BTW, you can use
command.local
to run
sleep
, which is how I solved for stuff not being ready in another codebase. I wouldn't call it ironclad, but it worked pretty reliably. Then you can make your Helm chart resource depend on the command.
That would normally work if I was installing CRDs independently from my Helm chart, but I would like the
Reelase
resource to install the required CRDs for that chart.
d

dry-keyboard-94795

09/07/2023, 8:53 PM
Is this your own chart or someone else? It's generally recommended to install CRDs independently of the chart, as helm doesn't really provide tooling for it (upgrades don't upgrade CRDs, for example).
f

full-eve-52536

09/07/2023, 8:54 PM
They are 3rd party charts, like
cert-manager
and
postgres
I guess
postgres
wouldn't have any CRDs, but you get the idea. I have several other 3rd party charts that I am trying to install via the
Release
resource which have CRDs
d

dry-keyboard-94795

09/07/2023, 9:09 PM
Without retry support, I'm not sure there's a direct workaround. If it helps, you can pass a URL to a crd manifest to
ConfigYaml
, which is how we deploy cert-manager internally
f

full-eve-52536

09/07/2023, 9:12 PM
Thank you for that tip!
d

dry-keyboard-94795

09/07/2023, 9:13 PM
I mean
yaml.ConfigFile
e

echoing-dinner-19531

09/08/2023, 7:56 AM
We do have an issue to add retries https://github.com/pulumi/pulumi/issues/7932 It's a little complicated and fairly niche (normally it's fine to just run
up
again) but feel free to vote and comment on that.