https://pulumi.com logo
#general
Title
# general
f

full-dress-10026

02/20/2019, 7:49 PM
I seem to consistently get this "failure" when running
pulumi up
with my `FargateService`:
Copy code
updating urn:pulumi:app-dev::app::awsx:x:ecs:FargateService$aws:ecs/service:Service::http-backend: timeout while waiting for state to become 'true' (last state: 'false', timeout: 10m0s)
Not sure what causes that because the service is correctly deployed and currently healthy. Any ideas?
l

lemon-spoon-91807

02/20/2019, 7:51 PM
No. Anything on the AWS side? Can you look at the logs there and see what's up?
can you link to the code that is creating the service?
f

full-dress-10026

02/20/2019, 7:53 PM
Anything on the AWS side?
Status looks ok
Can you look at the logs there and see what's up?
Which logs?
can you link to the code that is creating the service?
Copy code
let httpBackendService = new awsx.ecs.FargateService("http-backend", {
    cluster: cluster,
    taskDefinitionArgs: {
        containers: {
            httpBackend: {
                image: `<http://734247230719.dkr.ecr.us-west-2.amazonaws.com/compute/app-ions:${appIonsImageTag}|734247230719.dkr.ecr.us-west-2.amazonaws.com/compute/app-ions:${appIonsImageTag}>`,
                portMappings: [httpBackendListener],
                cpu: 1024,
                memory: 2048,
                environment: [
                    {
                        name: "COMPUTE_CONFIG_URI",
                        value: configUri
                    },
                    {
                        name: "NEW_RELIC_APP_NAME",
                        value: "name"
                    },
                    {
                        name: "NEW_RELIC_LICENSE_KEY",
                        value: "..."
                    },
                    {
                        name: "NEW_RELIC_ENVIRONMENT",
                        value: env
                    }
                ]
            }
        }
    },
    // the app seems to take 4 mins to start up
    healthCheckGracePeriodSeconds: 60 * 4,
    desiredCount: 2
});
l

lemon-spoon-91807

02/20/2019, 7:56 PM
Which logs?
looking to see what fargate might provide
f

full-dress-10026

02/20/2019, 7:56 PM
Here's the lb code:
Copy code
const httpBackendLb = new awsx.elasticloadbalancingv2.NetworkLoadBalancer("http-backend-lb3", {vpc: vpc});

const httpBackendTg = httpBackendLb.createTargetGroup("http-backend-tg2", {
    port: 8880,
    healthCheck: {
        path: "/_healthcheck",
        // healthyThreshold: 2,
        // unhealthyThreshold: 4
        // matcher: "200"
    }
});

const httpBackendListener = httpBackendTg.createListener("http-backend-listener", {
    port: 8880
});
Fargate has logs?
l

lemon-spoon-91807

02/20/2019, 7:56 PM
i would hope so 🙂
(if not... figuring tings out will suck!)
f

full-dress-10026

02/20/2019, 7:57 PM
How would I get to them? I see my application logs.
l

lemon-spoon-91807

02/20/2019, 7:59 PM
so, if you go to the aws console
then find your cluster:
then your service:
then there are logs:
f

full-dress-10026

02/20/2019, 8:01 PM
Oh, never seen that. It says "No results"
Wait, it just updated. It's showing my app logs.
My app logs look fine.
l

lemon-spoon-91807

02/20/2019, 8:02 PM
er...
so your service is running with no problems?
f

full-dress-10026

02/20/2019, 8:02 PM
Yep
l

lemon-spoon-91807

02/20/2019, 8:02 PM
what's under 'events'?
f

full-dress-10026

02/20/2019, 8:02 PM
Running and healthy
A bunch of this
l

lemon-spoon-91807

02/20/2019, 8:04 PM
that doesn't look good
it looks like it's trying to come up, then failing, then draining
f

full-dress-10026

02/20/2019, 8:05 PM
That looks like it's draining the old tasks.
l

lemon-spoon-91807

02/20/2019, 8:05 PM
why is it doing that though?
isn't that generally because something is wrong, and it's trying to bring things back up again?
f

full-dress-10026

02/20/2019, 8:05 PM
Because I am redeploying.
l

lemon-spoon-91807

02/20/2019, 8:11 PM
oh...
so if you look at the logs now, it's in a steady state?
f

full-dress-10026

02/20/2019, 8:12 PM
Yes. The last message is:
service http-backend-55db594 has reached a steady state.
l

lemon-spoon-91807

02/20/2019, 8:12 PM
hrmm... and how long did that take?
f

full-dress-10026

02/20/2019, 8:13 PM
From
service http-backend-55db594 has started 2 tasks
to
service http-backend-55db594 has reached a steady state.
took about 11 minutes.
l

lemon-spoon-91807

02/20/2019, 8:13 PM
so it may actually be hitting that 10m timeout
let me see if htat's controllable
f

full-dress-10026

02/20/2019, 8:13 PM
Can I disable the wait for healthy option?
l

lemon-spoon-91807

02/20/2019, 8:14 PM
i believe so. looking for the option
set
waitForSteadyState: false
👌 1
f

full-dress-10026

02/20/2019, 8:15 PM
For other uses, how would I set the timeout to a larger number?
l

lemon-spoon-91807

02/20/2019, 8:26 PM
I do not know 😕
it looks like it's just hardcoded to be 10min
as an aws api limitation