eager-cartoon-54068
04/15/2020, 4:33 PMerror: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:49758: connect: connection refused"
… panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::<project>::aws:batch/computeEnvironment:ComputeEnvironment::gpu-worker-computeEnvironment-us-east-1
last night. I destroyed the stack, rebooted, backed out all changes to a point that was definitely working. I don’t know what else to try. Any ideas? A change to an AWS Batch Compute Environment almost always results in a replacement.pulumi up -y
Previewing update (dev):
Type Name Plan Info
pulumi:pulumi:Stack PROJECT-dev 4 messages
~ ├─ aws:batch:ComputeEnvironment gpu-worker-computeEnvironment-us-east-1 update [diff: ~computeResources]
~ ├─ aws:batch:ComputeEnvironment gpu-watcher-computeEnvironment-us-east-1 update [diff: ~computeResources]
+- ├─ aws:batch:ComputeEnvironment cpu-watcher-computeEnvironment-us-east-1 replace [diff: ~computeResources]
+- ├─ aws:batch:ComputeEnvironment cpu-worker-computeEnvironment-us-east-1 replace [diff: ~computeResources]
~ ├─ aws:batch:JobQueue gpu-PROJECT-queue-us-east-1 update [diff: ~computeEnvironments]
~ ├─ aws:batch:JobQueue cpu-PROJECT-queue-us-east-1 update [diff: ~computeEnvironments]
~ ├─ aws:batch:ComputeEnvironment gpu-watcher-computeEnvironment-us-west-2 update [diff: ~computeResources]
+- ├─ aws:batch:ComputeEnvironment cpu-worker-computeEnvironment-us-west-2 replace [diff: ~computeResources]
+- ├─ aws:batch:ComputeEnvironment cpu-watcher-computeEnvironment-us-west-2 replace [diff: ~computeResources]
~ ├─ aws:batch:ComputeEnvironment gpu-worker-computeEnvironment-us-west-2 update [diff: ~computeResources]
~ ├─ aws:batch:JobQueue cpu-PROJECT-queue-us-west-2 update [diff: ~computeEnvironments]
~ └─ aws:batch:JobQueue gpu-PROJECT-queue-us-west-2 update [diff: ~computeEnvironments]
Diagnostics:
pulumi:pulumi:Stack (PROJECT-dev):
Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>
Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>
Updating (dev):
Type Name Status Info
pulumi:pulumi:Stack PROJECT-dev **failed** 1 error; 48 messages
+- ├─ aws:batch:ComputeEnvironment cpu-watcher-computeEnvironment-us-east-1 **replacing failed** [diff: ~computeResources]; 1 error
~ ├─ aws:batch:ComputeEnvironment gpu-worker-computeEnvironment-us-east-1 **updating failed** [diff: ~computeResources]; 1 error
~ ├─ aws:batch:ComputeEnvironment gpu-watcher-computeEnvironment-us-east-1 **updating failed** [diff: ~computeResources]; 1 error
+- ├─ aws:batch:ComputeEnvironment cpu-worker-computeEnvironment-us-east-1 **replacing failed** [diff: ~computeResources]; 1 error
~ ├─ aws:batch:ComputeEnvironment gpu-watcher-computeEnvironment-us-west-2 **updating failed** [diff: ~computeResources]; 1 error
+- ├─ aws:batch:ComputeEnvironment cpu-watcher-computeEnvironment-us-west-2 **replacing failed** [diff: ~computeResources]; 1 error
~ ├─ aws:batch:ComputeEnvironment gpu-worker-computeEnvironment-us-west-2 **updating failed** [diff: ~computeResources]; 1 error
+- └─ aws:batch:ComputeEnvironment cpu-worker-computeEnvironment-us-west-2 **replacing failed** [diff: ~computeResources]; 1 error
Diagnostics:
aws:batch:ComputeEnvironment (cpu-worker-computeEnvironment-us-east-1):
error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51376: connect: connection refused"
pulumi:pulumi:Stack (PROJECT-dev):
Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>
Synchronous call made to "aws:index/getAvailabilityZones:getAvailabilityZones" with an unregistered provider. This is now deprecated and may cause the program to hang.
For more details see: <https://www.pulumi.com/docs/troubleshooting/#synchronous-call>
panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::PROJECT::aws:batch/computeEnvironment:ComputeEnvironment::gpu-worker-computeEnvironment-us-east-1
goroutine 93 [running]:
<http://github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)|github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)>
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/failfast.go:23
<http://github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000b86c00|github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000b86c00>, 0x68e5b59, 0x48, 0xc001065728, 0x1, 0x1)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/assert.go:33 +0x197
<http://github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc0005c8480|github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc0005c8480>, 0x72201c0, 0xc001068bd0, 0xc0004b7490, 0xc0005c8480, 0x5c4ea01, 0xc00102ce80)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi-terraform-bridge@v1.8.5-0.20200402004340-25b33bb489e0/pkg/tfbridge/provider.go:825 +0x81d
<http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0>, 0xc001068bd0, 0x6643f80, 0xc0004b7490, 0x663a700, 0xacfe620, 0x72201c0, 0xc001068bd0)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1920 +0x89
<http://github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0|github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0>, 0xc00102bb90, 0x6643f80, 0xc0004b7490, 0xc000a4dec0, 0xc000a4dee0, 0x0, 0x0, 0x7169aa0, 0xc0003e71f0)
/home/travis/gopath/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2f9
<http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0>, 0xc0005c8480, 0x72201c0, 0xc00102bb90, 0xc0010c6660, 0xc0003bca60, 0x72201c0, 0xc00102bb90, 0xc000906a80, 0x97f)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1922 +0x14b
<http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc0007629c0|google.golang.org/grpc.(*Server).processUnaryRPC(0xc0007629c0>, 0x7244740, 0xc0005c8900, 0xc00100f800, 0xc0008ffd70, 0xacd0b38, 0x0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x4fd
<http://google.golang.org/grpc.(*Server).handleStream(0xc0007629c0|google.golang.org/grpc.(*Server).handleStream(0xc0007629c0>, 0x7244740, 0xc0005c8900, 0xc00100f800, 0x0)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xd23
<http://google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004fad50|google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004fad50>, 0xc0007629c0, 0x7244740, 0xc0005c8900, 0xc00100f800)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xbb
created by <http://google.golang.org/grpc.(*Server).serveStreams.func1|google.golang.org/grpc.(*Server).serveStreams.func1>
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1
panic: fatal: An assertion has failed: Expected diff to not require deletion or replacement during Update of urn:pulumi:dev::PROJECT::aws:batch/computeEnvironment:ComputeEnvironment::gpu-watcher-computeEnvironment-us-west-2
goroutine 268 [running]:
<http://github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)|github.com/pulumi/pulumi/sdk/go/common/util/contract.failfast(...)>
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/failfast.go:23
<http://github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000bdc100|github.com/pulumi/pulumi/sdk/go/common/util/contract.Assertf(0xc000bdc100>, 0x68e5b59, 0x48, 0xc000b3d728, 0x1, 0x1)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/go/common/util/contract/assert.go:33 +0x197
<http://github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc000573e00|github.com/pulumi/pulumi-terraform-bridge/pkg/tfbridge.(*Provider).Update(0xc000573e00>, 0x72201c0, 0xc000f8c4e0, 0xc0004a7ab0, 0xc000573e00, 0x5c4ea01, 0xc000f85740)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi-terraform-bridge@v1.8.5-0.20200402004340-25b33bb489e0/pkg/tfbridge/provider.go:825 +0x81d
<http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler.func1(0x72201c0>, 0xc000f8c4e0, 0x6643f80, 0xc0004a7ab0, 0x663a700, 0xacfe620, 0x72201c0, 0xc000f8c4e0)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1920 +0x89
<http://github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0|github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x72201c0>, 0xc000f79560, 0x6643f80, 0xc0004a7ab0, 0xc000c0e4c0, 0xc000c0e4e0, 0x0, 0x0, 0x7169aa0, 0xc0003e71f0)
/home/travis/gopath/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2f9
<http://github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0|github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Update_Handler(0x66ed4e0>, 0xc000573e00, 0x72201c0, 0xc000f79560, 0xc00005be60, 0xc0003bbda0, 0x72201c0, 0xc000f79560, 0xc000f8a000, 0x919)
/home/travis/gopath/pkg/mod/github.com/pulumi/pulumi/sdk@v1.14.1-0.20200402003901-9e377419164b/proto/go/provider.pb.go:1922 +0x14b
<http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc00076b040|google.golang.org/grpc.(*Server).processUnaryRPC(0xc00076b040>, 0x7244740, 0xc0004eb500, 0xc000b29700, 0xc00090bd70, 0xacd0b38, 0x0, 0x0, 0x0)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x4fd
<http://google.golang.org/grpc.(*Server).handleStream(0xc00076b040|google.golang.org/grpc.(*Server).handleStream(0xc00076b040>, 0x7244740, 0xc0004eb500, 0xc000b29700, 0x0)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xd23
<http://google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000930050|google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000930050>, 0xc00076b040, 0x7244740, 0xc0004eb500, 0xc000b29700)
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xbb
created by <http://google.golang.org/grpc.(*Server).serveStreams.func1|google.golang.org/grpc.(*Server).serveStreams.func1>
/home/travis/gopath/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1
error: update failed
aws:batch:ComputeEnvironment (gpu-worker-computeEnvironment-us-west-2):
error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51382: connect: connection refused"
aws:batch:ComputeEnvironment (gpu-watcher-computeEnvironment-us-east-1):
error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51376: connect: connection refused"
aws:batch:ComputeEnvironment (gpu-watcher-computeEnvironment-us-west-2):
error: transport is closing
aws:batch:ComputeEnvironment (gpu-worker-computeEnvironment-us-east-1):
error: transport is closing
aws:batch:ComputeEnvironment (cpu-worker-computeEnvironment-us-west-2):
error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51382: connect: connection refused"
aws:batch:ComputeEnvironment (cpu-watcher-computeEnvironment-us-east-1):
error: transport is closing
aws:batch:ComputeEnvironment (cpu-watcher-computeEnvironment-us-west-2):
error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:51382: connect: connection refused"
Resources:
131 unchanged
Duration: 8s
$ pulumi version
v1.14.1
$ cat package.json
{
"name": "PROJECT",
"devDependencies": {
"@types/node": "^8.10.60"
},
"dependencies": {
"@pulumi/aws": "^1.31.0",
"@pulumi/awsx": "^0.18.10",
"@pulumi/pulumi": "^1.14.1"
}
}
gentle-diamond-70147
04/15/2020, 5:06 PMeager-cartoon-54068
04/15/2020, 5:10 PMgentle-diamond-70147
04/15/2020, 7:39 PMeager-cartoon-54068
04/15/2020, 7:54 PMExpected diff to not require deletion or replacement during UpdateI figured I could make the error go away if I could tell pulumi to expect replacement and not an incremental update.
deleteBeforeReplace
will workc3.xlarge
is fine in us-west-2
, but not in us-east-2
. I’m experimenting with various instance types so I occasionally add one that isn’t supported in all regions. When I do that, pulumi’s state gets screwed up(re: ☝️) and I have to destroy the stack and remove the offending instance type to get things into a good state.
recap:
• deploy CE with c5, good
• deploy CE with c3, c5 - update kicked back with invalid type for us-east-2
• remove c3 and deploy with original c5 - transport error I posted above and no obvious way to recover without destroying everythinggentle-diamond-70147
04/15/2020, 8:19 PMeager-cartoon-54068
04/15/2020, 8:54 PMgentle-diamond-70147
04/16/2020, 3:38 PMeager-cartoon-54068
04/16/2020, 4:16 PM