I seem to consistently get this "failure" when run...
# general
f
I seem to consistently get this "failure" when running
pulumi up
with my `FargateService`:
Copy code
updating urn:pulumi:app-dev::app::awsx:x:ecs:FargateService$aws:ecs/service:Service::http-backend: timeout while waiting for state to become 'true' (last state: 'false', timeout: 10m0s)
Not sure what causes that because the service is correctly deployed and currently healthy. Any ideas?
l
No. Anything on the AWS side? Can you look at the logs there and see what's up?
can you link to the code that is creating the service?
f
Anything on the AWS side?
Status looks ok
Can you look at the logs there and see what's up?
Which logs?
can you link to the code that is creating the service?
Copy code
let httpBackendService = new awsx.ecs.FargateService("http-backend", {
    cluster: cluster,
    taskDefinitionArgs: {
        containers: {
            httpBackend: {
                image: `<http://734247230719.dkr.ecr.us-west-2.amazonaws.com/compute/app-ions:${appIonsImageTag}|734247230719.dkr.ecr.us-west-2.amazonaws.com/compute/app-ions:${appIonsImageTag}>`,
                portMappings: [httpBackendListener],
                cpu: 1024,
                memory: 2048,
                environment: [
                    {
                        name: "COMPUTE_CONFIG_URI",
                        value: configUri
                    },
                    {
                        name: "NEW_RELIC_APP_NAME",
                        value: "name"
                    },
                    {
                        name: "NEW_RELIC_LICENSE_KEY",
                        value: "..."
                    },
                    {
                        name: "NEW_RELIC_ENVIRONMENT",
                        value: env
                    }
                ]
            }
        }
    },
    // the app seems to take 4 mins to start up
    healthCheckGracePeriodSeconds: 60 * 4,
    desiredCount: 2
});
l
Which logs?
looking to see what fargate might provide
f
Here's the lb code:
Copy code
const httpBackendLb = new awsx.elasticloadbalancingv2.NetworkLoadBalancer("http-backend-lb3", {vpc: vpc});

const httpBackendTg = httpBackendLb.createTargetGroup("http-backend-tg2", {
    port: 8880,
    healthCheck: {
        path: "/_healthcheck",
        // healthyThreshold: 2,
        // unhealthyThreshold: 4
        // matcher: "200"
    }
});

const httpBackendListener = httpBackendTg.createListener("http-backend-listener", {
    port: 8880
});
Fargate has logs?
l
i would hope so 🙂
(if not... figuring tings out will suck!)
f
How would I get to them? I see my application logs.
l
so, if you go to the aws console
then find your cluster:
then your service:
then there are logs:
f
Oh, never seen that. It says "No results"
Wait, it just updated. It's showing my app logs.
My app logs look fine.
l
er...
so your service is running with no problems?
f
Yep
l
what's under 'events'?
f
Running and healthy
A bunch of this
l
that doesn't look good
it looks like it's trying to come up, then failing, then draining
f
That looks like it's draining the old tasks.
l
why is it doing that though?
isn't that generally because something is wrong, and it's trying to bring things back up again?
f
Because I am redeploying.
l
oh...
so if you look at the logs now, it's in a steady state?
f
Yes. The last message is:
service http-backend-55db594 has reached a steady state.
l
hrmm... and how long did that take?
f
From
service http-backend-55db594 has started 2 tasks
to
service http-backend-55db594 has reached a steady state.
took about 11 minutes.
l
so it may actually be hitting that 10m timeout
let me see if htat's controllable
f
Can I disable the wait for healthy option?
l
i believe so. looking for the option
set
waitForSteadyState: false
👌 1
f
For other uses, how would I set the timeout to a larger number?
l
I do not know 😕
it looks like it's just hardcoded to be 10min
as an aws api limitation