Is there a way to tell pulumi to retry before fail...
# dotnetw
wet-noon-14291
01/30/2021, 1:03 AMIs there a way to tell pulumi to retry before failing when creating a resource. I have some trouble creating an API Management thing that seems timing related, so 1-2 retry should fix it. I have dependencies, but that doesn't seem to help for some reason.
t
tall-librarian-49374
01/30/2021, 7:51 AMNormally, we do retries automatically, if they are warranted based on the HTTP code that we get
tall-librarian-49374
01/30/2021, 7:52 AMWhich error do you get and what’s the timing?
w
wet-noon-14291
01/30/2021, 9:01 AMWill check more in detail later tonight. It is when updating APIM and sometime it seems like it get a 40x against the openapi endpoint in the app. I’m pretty sure it would succeed if it just tried again in a couple of seconds.
wet-noon-14291
01/30/2021, 9:01 AMIt doesn’t happen every time and the app is up right after the error has occurred.
wet-noon-14291
01/30/2021, 9:02 AMIs it there I could use the customTimeouts?
t
tall-librarian-49374
01/30/2021, 9:20 AMCustom timeouts won’t help… 40x is supposed to mean a client error which doesn’t normally make sense to retry.
w
wet-noon-14291
01/30/2021, 11:03 PMWe have the following setup that we deploy with pulumi (simplified)
* App (.NET Core 5) as kubernetes deployment
* Traefik ingress
* APIM
The APIM API resource depends on the kubernetes deployment resource and the ingress resource. In most deploys that failed we did no changes to the ingress, only the deployment. The kubernetes deploy succeed, but then the APIM API deploy fails. Error message is 503 (not 40x as I mentioned), when APIM is trying to get the openapi spec. Not sure why it is 503, since the deploy of kubernetes deployment is finished and we have configured liveness, startup and readiness probe setup.
t
tall-librarian-49374
01/31/2021, 7:56 AMSo, API Management gets an error when trying to read the spec from k8s, gets 503, and then pushes down the error to Pulumi?
w
wet-noon-14291
01/31/2021, 9:34 AM@tall-librarian-49374 That’s how I read the log output. I can share it later today.
wet-noon-14291
01/31/2021, 12:12 PMThe last part of the logs from one of the failed runs:
Copy code
~ azure:apimanagement:Api ApimApiExternal updating [diff: ~import]
~ azure:apimanagement:Api ApimApiInternal updating [diff: ~import]
~ azure:apimanagement:Api ApimApiExternal updating [diff: ~import]; error: 1 error occurred:
~ azure:apimanagement:Api ApimApiExternal **updating failed** [diff: ~import]; error: 1 error occurred:
~ azure:apimanagement:Api ApimApiInternal updated [diff: ~import]
pulumi:pulumi:Stack XXXXX.yyyyyyy.zzzzzzz.Deploy-XXXXX.yyyyyyy.zzzzzzz.deploy.dev running error: update failed
pulumi:pulumi:Stack XXXXX.yyyyyyy.zzzzzzz.Deploy-XXXXX.yyyyyyy.zzzzzzz.deploy.dev **failed** 1 error
Diagnostics:
azure:apimanagement:Api (ApimApiExternal):
error: 1 error occurred:
* updating urn:pulumi:XXXXX.yyyyyyy.zzzzzzz.deploy.dev::XXXXX.yyyyyyy.zzzzzzz.Deploy::azure:apimanagement/api:Api::ApimApiExternal: creating/updating API Management API "XXXXX-yyyyyyy-zzzzzzz-api-external" (Resource Group "elkds-dev-cmn-rg"): apimanagement.APIClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ValidationError" Message="One or more fields contain incorrect values:" Details=[{"code":"ValidationError","message":"Parsing error(s): Failed to import from specified resource <https://zzzzzzz.elkds-dev.XXXXX.com/external/api/swagger/v1/swagger.json?guid=db534bd4-1e11-4590-9c50-c5f09d011921>: Response status code does not indicate success: 503 (Service Unavailable)..","target":"representation"}]
pulumi:pulumi:Stack (XXXXX.yyyyyyy.zzzzzzz.Deploy-XXXXX.yyyyyyy.zzzzzzz.deploy.dev):
error: update failedt
tall-librarian-49374
02/01/2021, 7:08 AMMaybe shoot a question in #kubernetes - I feel like it’s more related to the await logic there?
w
wet-noon-14291
02/01/2021, 6:55 PMYeah. It could also be our probing function that returns ok before the swagger is up or something.
wet-noon-14291
02/02/2021, 11:07 AMOn a second though, I don't understand how that could happen since the app needs to be up for the probing to return ok since it is http probing. Maybe there is some timing issue when traefik start directing to the new pods.
wet-noon-14291
02/05/2021, 7:30 PMI did modify the startup probe so it is making a request to the swagger endpoint to make sure that is responding, that seems to have fixed it.
wet-noon-14291
02/08/2021, 9:01 PMI thought this actually fixed it, but apparently not. We still see some weird 503 and 502 errors when APIM is being updated. The 503 and 502 is when APIM is reaching out to get the openapi spec.
6 Views