Channels
announcements
automation-api
aws
azure
blog-posts
built-with-pulumi
cloudengineering
cloudengineering-support
content-share
contribex
contribute
docs
dotnet
finops
general
getting-started
gitlab
golang
google-cloud
hackathon-03-19-2020
hacktoberfest
install
java
jobs
kubernetes
learn-pulumi-events
linen
localstack
multi-language-hackathon
office-hours
oracle-cloud-infrastructure
plugin-framework
pulumi-ai
pulumi-cdk
pulumi-cloud
pulumi-crosscode
pulumi-deployments
pulumi-kubernetes-operator
pulumiverse
python
registry
status
testingtesting123
testingtesting321
typescript
welcome
workshops
yaml
Title
w
wet-noon-14291
01/30/2021, 1:03 AMIs there a way to tell pulumi to retry before failing when creating a resource. I have some trouble creating an API Management thing that seems timing related, so 1-2 retry should fix it. I have dependencies, but that doesn't seem to help for some reason.
t
tall-librarian-49374
01/30/2021, 7:51 AMNormally, we do retries automatically, if they are warranted based on the HTTP code that we get
Which error do you get and what’s the timing?
w
wet-noon-14291
01/30/2021, 9:01 AMWill check more in detail later tonight. It is when updating APIM and sometime it seems like it get a 40x against the openapi endpoint in the app. I’m pretty sure it would succeed if it just tried again in a couple of seconds.
It doesn’t happen every time and the app is up right after the error has occurred.
Is it there I could use the customTimeouts?
t
tall-librarian-49374
01/30/2021, 9:20 AMCustom timeouts won’t help… 40x is supposed to mean a client error which doesn’t normally make sense to retry.
w
wet-noon-14291
01/30/2021, 11:03 PMWe have the following setup that we deploy with pulumi (simplified)
* App (.NET Core 5) as kubernetes deployment
* Traefik ingress
* APIM
The APIM API resource depends on the kubernetes deployment resource and the ingress resource. In most deploys that failed we did no changes to the ingress, only the deployment. The kubernetes deploy succeed, but then the APIM API deploy fails. Error message is 503 (not 40x as I mentioned), when APIM is trying to get the openapi spec. Not sure why it is 503, since the deploy of kubernetes deployment is finished and we have configured liveness, startup and readiness probe setup.
t
tall-librarian-49374
01/31/2021, 7:56 AMSo, API Management gets an error when trying to read the spec from k8s, gets 503, and then pushes down the error to Pulumi?
w
wet-noon-14291
01/31/2021, 9:34 AM@tall-librarian-49374 That’s how I read the log output. I can share it later today.
The last part of the logs from one of the failed runs:
~ azure:apimanagement:Api ApimApiExternal updating [diff: ~import]
~ azure:apimanagement:Api ApimApiInternal updating [diff: ~import]
~ azure:apimanagement:Api ApimApiExternal updating [diff: ~import]; error: 1 error occurred:
~ azure:apimanagement:Api ApimApiExternal **updating failed** [diff: ~import]; error: 1 error occurred:
~ azure:apimanagement:Api ApimApiInternal updated [diff: ~import]
pulumi:pulumi:Stack XXXXX.yyyyyyy.zzzzzzz.Deploy-XXXXX.yyyyyyy.zzzzzzz.deploy.dev running error: update failed
pulumi:pulumi:Stack XXXXX.yyyyyyy.zzzzzzz.Deploy-XXXXX.yyyyyyy.zzzzzzz.deploy.dev **failed** 1 error
Diagnostics:
azure:apimanagement:Api (ApimApiExternal):
error: 1 error occurred:
* updating urn:pulumi:XXXXX.yyyyyyy.zzzzzzz.deploy.dev::XXXXX.yyyyyyy.zzzzzzz.Deploy::azure:apimanagement/api:Api::ApimApiExternal: creating/updating API Management API "XXXXX-yyyyyyy-zzzzzzz-api-external" (Resource Group "elkds-dev-cmn-rg"): apimanagement.APIClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ValidationError" Message="One or more fields contain incorrect values:" Details=[{"code":"ValidationError","message":"Parsing error(s): Failed to import from specified resource <https://zzzzzzz.elkds-dev.XXXXX.com/external/api/swagger/v1/swagger.json?guid=db534bd4-1e11-4590-9c50-c5f09d011921>: Response status code does not indicate success: 503 (Service Unavailable)..","target":"representation"}]
pulumi:pulumi:Stack (XXXXX.yyyyyyy.zzzzzzz.Deploy-XXXXX.yyyyyyy.zzzzzzz.deploy.dev):
error: update failedt
tall-librarian-49374
02/01/2021, 7:08 AMMaybe shoot a question in #kubernetes - I feel like it’s more related to the await logic there?
w
wet-noon-14291
02/01/2021, 6:55 PMYeah. It could also be our probing function that returns ok before the swagger is up or something.
On a second though, I don't understand how that could happen since the app needs to be up for the probing to return ok since it is http probing. Maybe there is some timing issue when traefik start directing to the new pods.
I did modify the startup probe so it is making a request to the swagger endpoint to make sure that is responding, that seems to have fixed it.
I thought this actually fixed it, but apparently not. We still see some weird 503 and 502 errors when APIM is being updated. The 503 and 502 is when APIM is reaching out to get the openapi spec.