It's there any movement on this issue? It is quite easy to accidentally bring an app down. Quite scary that this is possible e.g. by adding additional permissions to a service running on ECS. This seems like quite basic functionality? Has anyone here running on ECS and deploying using Pulumi experienced this? Any workarounds at all?

Can you create the same policy twice so there's no shared attachment?

What do you mean? 🙂

Is the policy doc: 1. Customer-managed and controlled by Pulumi 2. Customer-managed and one you are looking up e.g. via

.get()

3. An AWS-managed policy

Untitled.ts

Side note: You might wanna move the inline policies to separate customer-managed policies because you can quickly hit a limit on IAM policy length.

Possible that the issue I've attached is not relevant, but I've definitely experienced bringing my app down a few times now due to Pulumi deleting policy attachments.

Ok, that's definitely not good. Do you know any more details about under what conditions that outage happens? The most useful thing here would be the detailed diff (

details

) when

pulumi up

shows the preview.

The last time it happened (this morning), this was the order of events. Run pulumi refresh (shortened):

~   │  └─ aws:ecs:Service                         blueberry                                                   updated (0.00s)     [diff: ~taskDefinition]
~   │  └─ aws:ecs:TaskDefinition                  blueberry                                                   updated (0.00s)     [diff: ~containerDefinitions]
~   ├─ aws:ec2:Eip                                shared-prod-eip-2                                           updated (1s)        [diff: ~associationId,instance,networkInterface,privateDns,privateIp]
~   ├─ aws:ec2:Eip                                shared-prod-eip-0                                           updated (1s)        [diff: ~associationId,instance,networkInterface,privateDns,privateIp]
 ~   ├─ docker-build:index:Image                   blueberry                                                   updated (5s)        [diff: ~contextHash]
 ~   ├─ aws:iam:Role                               blueberry-execution                                         updated (5s)        [diff: ~managedPolicyArns]
~   ├─ aws:ec2:Eip                                shared-prod-eip-1                                           updated (1s)        [diff: ~associationId,instance,networkInterface,privateDns,privateIp]

Resources:
    ~ 10 updated
    143 unchanged

Pulumi update (shortened):

~   ├─ docker-build:index:Image       blueberry            updated          [diff: ~contextHash]
 ~   └─ aws:iam:Role                   blueberry-execution  updated (7s)     [diff: ~managedPolicyArns]

Outputs:
  - policyAttachments: [
  -     [0]: "arn:aws:iam::xxx:policy/kcl-production-xxx"
  -     [4]: "arn:aws:iam::xxx:policy/dynamodb-table-policy-production-xxx"
  -     [9]: "arn:aws:iam::xxx:policy/invoke-all-lambdas"
  -     [10]: "arn:aws:iam::xxx:policy/cognito-permissions"
    ]
  - serviceName      : "blueberry"
  - streamNames      : [
  -     [0]: "xxx-stream"
    ]

I think when there's a diff on

managedPolicyArns

it then proceeds to delete the attachments. The attachments then get recreated if I run

pulumi update

again.

Can you get the detailed diff next time you do an update? It'll show you exactly what attributes changed and from what to what. I don't see a recreate in there, so that'll probably not what's happening.

When I do an update, normally the only thing that changes is:

~   ├─ docker-build:index:Image       blueberry            updated          [diff: ~contextHash]

pulumi preview --diff

will give you the detailed diff. You can also select

details

when you run

pulumi up

to get the same output.

This is quite easily reproducible by running refresh followed by an update