What do you do when Pulumi seems to have lost all connection to the infra its created? I'm seeing subnets created, security groups, etc. and

pulumi refresh

is not reconnecting the state to the running infrastructure.

pulumi stack export

also lists the resources that it created and I'm not immediately seeing any discrepancies. Thanks.

Can you describe what you're seeing that looks wrong? Is there a timeout? Are your credentials expired or wrong?

It sits there trying to delete the security group and role before trying to delete the sagemaker instance. Those deletions keep going for more than 900s.

It hasn't lost connection. Maybe the credentials expired just before the deletion attempt was made, but more likely, something that isn't managed by Pulumi has started to use the security group. So AWS attempts to action the delete, but it won't happen.

@little-cartoon-10569 thanks for that suggestion! The problem was that there was an ENI created that Pulumi did not have in the state file so neither the subnet nor the security group could be deleted. Now, I'm not sure why Pulumi created a network interface that was not in the state file, but I will keep an eye on that to see if it happens again.

Pulumi won't have created it, AWS did. Any time you (or Pulumi) creates an addressable resource like an EC2 instance, an EBS storage device, or a load balancer, it creates at least one ENI for it. Generally they disappear when the associated resource does. Can't say why it didn't disappear this time, and it's probably not possible to figure it out from CloadWatch based on a single occurrence :( Glad it's sorted!