I have a Pulumi configuration to deploy something ...
# aws
m
I have a Pulumi configuration to deploy something onto fargate using
awsx
. However, when the service errors, AWS simply keeps trying to deploy the same service and Pulumi hangs indefinitely. How can I get Pulumi to fail on a failed deploy instead of having AWS retry endlessly?
Would like to have a reliable way for pulumi to attempt to deploy, fail, and rollback. Wondering if there is prior art here as I don't see anything in the docs.
s
Pulumi should never hang indefinitely. In some cases, it may take something like 10 to 15 mins to fail, but this is common among all IaC tools to my knowledge.
For example, if a security group cannot be deleted because it's in use, it'll take about that long to fail.
m
I waited a little over 22 minutes and ended up killing the deploy myself. When I checked the AWS ECS task logs, it looks like it tried to launch the task multiple times failing each time. Is there a way to tell it to fail fast (if the first deploy fails, fail on pulumi)?
22 minutes is an awfully long time to wait to see if a deploy failed.
m
I've pasted examples of setting up an ECS Fargate cluster.
q
@mysterious-family-43099 This is more of an ECS issue than a Pulumi issue, most likely. ECS will continue to try to get the desired count of the service. Have you enabled the deployment circuit breaker on the ECS Fargate service? Also, if you have a load balanced service, the health check settings on the load balancer will come into play as well. You can get into the same kind of situation with CloudFormation as well, so it is not specific to Pulumi.
m
@quaint-hydrogen-7228 I saw the deployment circuit breaker. I am going to try to enable that. I think my load balancer does 3 health checks before making sure the service is healthy.
@quaint-hydrogen-7228 Looks like the deployment circuit breaker needs the deployment to fail 10 times before it triggers a rollback.. This threshold also appears to not be configurable. Is there a better way to do this other than the deployment circuitbreaker? @millions-furniture-75402 Sorry I don't see the examples. Where are they? Hoping to just figure out a way to do fail fast and rollback so failed deploys don't break things and move quickly.