https://pulumi.com logo
Powered by
e

eager-cartoon-54068

04/15/2020, 4:33 PM
I started getting 
error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:49758: connect: connection refused"
… 
panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::<project>::aws:batch/computeEnvironment:ComputeEnvironment::gpu-worker-computeEnvironment-us-east-1
last night. I destroyed the stack, rebooted, backed out all changes to a point that was definitely working. I don’t know what else to try. Any ideas? A change to an AWS Batch Compute Environment almost always results in a replacement.
I did upgrade my pulumi client and pulumi typescript dependencies just before this started happening.
Copy code
pulumi up -y
Previewing update (dev):
     Type                             Name                                      Plan        Info
     pulumi:pulumi:Stack              PROJECT-dev                                4 messages
 ~   ├─ aws:batch:ComputeEnvironment  gpu-worker-computeEnvironment-us-east-1   update      [diff: ~computeResources]
 ~   ├─ aws:batch:ComputeEnvironment  gpu-watcher-computeEnvironment-us-east-1  update      [diff: ~computeResources]
 +-  ├─ aws:batch:ComputeEnvironment  cpu-watcher-computeEnvironment-us-east-1  replace     [diff: ~computeResources]
 +-  ├─ aws:batch:ComputeEnvironment  cpu-worker-computeEnvironment-us-east-1   replace     [diff: ~computeResources]
 ~   ├─ aws:batch:JobQueue            gpu-PROJECT-queue-us-east-1               update      [diff: ~computeEnvironments]
 ~   ├─ aws:batch:JobQueue            cpu-PROJECT-queue-us-east-1               update      [diff: ~computeEnvironments]
 ~   ├─ aws:batch:ComputeEnvironment  gpu-watcher-computeEnvironment-us-west-2  update      [diff: ~computeResources]
 +-  ├─ aws:batch:ComputeEnvironment  cpu-worker-computeEnvironment-us-west-2   replace     [diff: ~computeResources]
 +-  ├─ aws:batch:ComputeEnvironment  cpu-watcher-computeEnvironment-us-west-2  replace     [diff: ~computeResources]
 ~   ├─ aws:batch:ComputeEnvironment  gpu-worker-computeEnvironment-us-west-2   update      [diff: ~computeResources]
 ~   ├─ aws:batch:JobQueue            cpu-PROJECT-queue-us-west-2               update      [diff: ~computeEnvironments]
 ~   └─ aws:batch:JobQueue            gpu-PROJECT-queue-us-west-2               update      [diff: ~computeEnvironments]

Diagnostics:
  pulumi:pulumi:Stack (PROJECT-dev):
    Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
    For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>
    Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
    For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>


Updating (dev):
     Type                             Name                                      Status                   Info
     pulumi:pulumi:Stack              PROJECT-dev                    **failed**               1 error; 48 messages
 +-  ├─ aws:batch:ComputeEnvironment  cpu-watcher-computeEnvironment-us-east-1  **replacing failed**     [diff: ~computeResources]; 1 error
 ~   ├─ aws:batch:ComputeEnvironment  gpu-worker-computeEnvironment-us-east-1   **updating failed**      [diff: ~computeResources]; 1 error
 ~   ├─ aws:batch:ComputeEnvironment  gpu-watcher-computeEnvironment-us-east-1  **updating failed**      [diff: ~computeResources]; 1 error
 +-  ├─ aws:batch:ComputeEnvironment  cpu-worker-computeEnvironment-us-east-1   **replacing failed**     [diff: ~computeResources]; 1 error
 ~   ├─ aws:batch:ComputeEnvironment  gpu-watcher-computeEnvironment-us-west-2  **updating failed**      [diff: ~computeResources]; 1 error
 +-  ├─ aws:batch:ComputeEnvironment  cpu-watcher-computeEnvironment-us-west-2  **replacing failed**     [diff: ~computeResources]; 1 error
 ~   ├─ aws:batch:ComputeEnvironment  gpu-worker-computeEnvironment-us-west-2   **updating failed**      [diff: ~computeResources]; 1 error
 +-  └─ aws:batch:ComputeEnvironment  cpu-worker-computeEnvironment-us-west-2   **replacing failed**     [diff: ~computeResources]; 1 error

Diagnostics:
  aws:batch:ComputeEnvironment (cpu-worker-computeEnvironment-us-east-1):
    error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51376: connect: connection refused"

  pulumi:pulumi:Stack (PROJECT-dev):
    Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
    For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>
    Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
    For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>

    panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::PROJECT::aws:batch/computeEnvironment:ComputeEnvironment::gpu-worker-computeEnvironment-us-east-1
    goroutine 93 [running]:
    <http://github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)|github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)>
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/failfast.go:23
    <http://github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000b86c00|github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000b86c00>, 0x68e5b59, 0x48, 0xc001065728, 0x1, 0x1)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/assert.go:33 +0x197
    <http://github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc0005c8480|github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc0005c8480>, 0x72201c0, 0xc001068bd0, 0xc0004b7490, 0xc0005c8480, 0x5c4ea01, 0xc00102ce80)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi-terraform-bridge@v1.8.5-0.20200402004340-25b33bb489e0/pkg/tfbridge/provider.go:825 +0x81d
    <http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0>, 0xc001068bd0, 0x6643f80, 0xc0004b7490, 0x663a700, 0xacfe620, 0x72201c0, 0xc001068bd0)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1920 +0x89
    <http://github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0|github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0>, 0xc00102bb90, 0x6643f80, 0xc0004b7490, 0xc000a4dec0, 0xc000a4dee0, 0x0, 0x0, 0x7169aa0, 0xc0003e71f0)
        /home/travis/gopath/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2f9
    <http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0>, 0xc0005c8480, 0x72201c0, 0xc00102bb90, 0xc0010c6660, 0xc0003bca60, 0x72201c0, 0xc00102bb90, 0xc000906a80, 0x97f)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1922 +0x14b
    <http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc0007629c0|google.golang.org/grpc.(*Server).processUnaryRPC(0xc0007629c0>, 0x7244740, 0xc0005c8900, 0xc00100f800, 0xc0008ffd70, 0xacd0b38, 0x0, 0x0, 0x0)
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x4fd
    <http://google.golang.org/grpc.(*Server).handleStream(0xc0007629c0|google.golang.org/grpc.(*Server).handleStream(0xc0007629c0>, 0x7244740, 0xc0005c8900, 0xc00100f800, 0x0)
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xd23
    <http://google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004fad50|google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004fad50>, 0xc0007629c0, 0x7244740, 0xc0005c8900, 0xc00100f800)
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xbb
    created by <http://google.golang.org/grpc.(*Server).serveStreams.func1|google.golang.org/grpc.(*Server).serveStreams.func1>
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1

    panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::PROJECT::aws:batch/computeEnvironment:ComputeEnvironment::gpu-watcher-computeEnvironment-us-west-2
    goroutine 268 [running]:
    <http://github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)|github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)>
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/failfast.go:23
    <http://github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000bdc100|github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000bdc100>, 0x68e5b59, 0x48, 0xc000b3d728, 0x1, 0x1)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/assert.go:33 +0x197
    <http://github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc000573e00|github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc000573e00>, 0x72201c0, 0xc000f8c4e0, 0xc0004a7ab0, 0xc000573e00, 0x5c4ea01, 0xc000f85740)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi-terraform-bridge@v1.8.5-0.20200402004340-25b33bb489e0/pkg/tfbridge/provider.go:825 +0x81d
    <http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0>, 0xc000f8c4e0, 0x6643f80, 0xc0004a7ab0, 0x663a700, 0xacfe620, 0x72201c0, 0xc000f8c4e0)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1920 +0x89
    <http://github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0|github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0>, 0xc000f79560, 0x6643f80, 0xc0004a7ab0, 0xc000c0e4c0, 0xc000c0e4e0, 0x0, 0x0, 0x7169aa0, 0xc0003e71f0)
        /home/travis/gopath/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2f9
    <http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0>, 0xc000573e00, 0x72201c0, 0xc000f79560, 0xc00005be60, 0xc0003bbda0, 0x72201c0, 0xc000f79560, 0xc000f8a000, 0x919)
        /home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1922 +0x14b
    <http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc00076b040|google.golang.org/grpc.(*Server).processUnaryRPC(0xc00076b040>, 0x7244740, 0xc0004eb500, 0xc000b29700, 0xc00090bd70, 0xacd0b38, 0x0, 0x0, 0x0)
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x4fd
    <http://google.golang.org/grpc.(*Server).handleStream(0xc00076b040|google.golang.org/grpc.(*Server).handleStream(0xc00076b040>, 0x7244740, 0xc0004eb500, 0xc000b29700, 0x0)
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xd23
    <http://google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000930050|google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000930050>, 0xc00076b040, 0x7244740, 0xc0004eb500, 0xc000b29700)
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xbb
    created by <http://google.golang.org/grpc.(*Server).serveStreams.func1|google.golang.org/grpc.(*Server).serveStreams.func1>
        /home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1

    error: update failed

  aws:batch:ComputeEnvironment (gpu-worker-computeEnvironment-us-west-2):
    error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51382: connect: connection refused"

  aws:batch:ComputeEnvironment (gpu-watcher-computeEnvironment-us-east-1):
    error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51376: connect: connection refused"

  aws:batch:ComputeEnvironment (gpu-watcher-computeEnvironment-us-west-2):
    error: transport is closing

  aws:batch:ComputeEnvironment (gpu-worker-computeEnvironment-us-east-1):
    error: transport is closing

  aws:batch:ComputeEnvironment (cpu-worker-computeEnvironment-us-west-2):
    error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51382: connect: connection refused"

  aws:batch:ComputeEnvironment (cpu-watcher-computeEnvironment-us-east-1):
    error: transport is closing

  aws:batch:ComputeEnvironment (cpu-watcher-computeEnvironment-us-west-2):
    error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51382: connect: connection refused"

Resources:
    131 unchanged

Duration: 8s
Copy code
$ pulumi version
v1.14.1
$ cat package.json 
{
    "name": "PROJECT",
    "devDependencies": {
        "@types/node": "^8.10.60"
    },
    "dependencies": {
        "@pulumi/aws": "^1.31.0",
        "@pulumi/awsx": "^0.18.10",
        "@pulumi/pulumi": "^1.14.1"
    }
}
According to https://github.com/pulumi/pulumi/issues/2914 the bug has been fixed 🤷‍♂️
g

gentle-diamond-70147

04/15/2020, 5:06 PM
While similar message I think that's a different bug. Do you mind opening a new issue with your info?
👍 1
e

eager-cartoon-54068

04/15/2020, 5:10 PM
is there a way to specify that a resource requires replacement?
g

gentle-diamond-70147

04/15/2020, 7:39 PM
Can you elaborate?
e

eager-cartoon-54068

04/15/2020, 7:54 PM
Expected diff to not require deletion or replacement during Update
I figured I could make the error go away if I could tell pulumi to expect replacement and not an incremental update.
actually, maybe 
deleteBeforeReplace
will work
I’ve narrowed down the issue. I’m using multiple providers to deploy AWS Batch Compute Environments to different AWS regions. Batch CEs doesn’t support all EC2 instances types in all regions so 
c3.xlarge
is fine in 
us-west-2
, but not in 
us-east-2
. I’m experimenting with various instance types so I occasionally add one that isn’t supported in all regions. When I do that, pulumi’s state gets screwed up(re: ☝️) and I have to destroy the stack and remove the offending instance type to get things into a good state. recap: • deploy CE with c5, good • deploy CE with c3, c5 - update kicked back with invalid type for us-east-2 • remove c3 and deploy with original c5 - transport error I posted above and no obvious way to recover without destroying everything
g

gentle-diamond-70147

04/15/2020, 8:19 PM
Oh, that's super helpful.
e

eager-cartoon-54068

04/15/2020, 8:54 PM
I have to slightly modify my last response. It takes a little fiddling to get it into a bad state, and now it seems to only show the transport error one time after a bad instance is introduced. I’ll keep digging.
g

gentle-diamond-70147

04/16/2020, 3:38 PM
When you get to a reproducible bit of code, please do open an issue with it. We definitely want to improve this experience, at least with more helpful error messages.
e

eager-cartoon-54068

04/16/2020, 4:16 PM
Will do
3 Views
Powered by