https://pulumi.com logo
Title
w

witty-helmet-74677

08/27/2021, 9:31 AM
Hi, during the pulumi destroy of a stack with an AWS S3 Bucket which has 15K objects takes around 45 minutes to delete the objects before deleting the bucket. Using AWS CLI the deletion takes 2 minutes. Is there any argument which could speed the stack destroy in this case or this is a know issue? Language: go.
b

busy-honey-73811

08/27/2021, 10:00 AM
Can confirm, we have the same problem with very large S3 Buckets and it takes literally hours to delete them with pulumi. Usually those stacks fail (timeout) on destroy and we have to manually clean-up the S3 Buckets (using CLI or AWS Web Console) to speed things up.
w

witty-helmet-74677

08/27/2021, 11:34 AM
One more addition. The objects are not big 2-3 KB each. In total like 40MB.
b

busy-honey-73811

08/27/2021, 11:49 AM
Same here, we have thousands (if not millions) of small to medium sized objects (KB to few MB).
g

great-sunset-355

08/27/2021, 2:04 PM
Do you manage each object with pulumi or did you try
force_destroy=True
?
b

billowy-army-68599

08/27/2021, 4:38 PM
could you run a performance trace here? https://www.pulumi.com/docs/troubleshooting/#performance
b

busy-honey-73811

08/27/2021, 6:57 PM
In my case we only manage the S3 Bucket with Pulumi, the objects are created by applications (not Pulumi).
w

witty-helmet-74677

08/30/2021, 8:37 AM
We also maintain only the bucket with pulumi. The objects are created by apps. The pulumi uses force destroy, otherwise wouldn't be able to delete the bucket.
I executed the pulumi destroy now with the traces turned on suggested by @billowy-army-68599. The 45 minutes are spent in /pulumirpc.ResourceProvider/Delete.
b

billowy-army-68599

08/30/2021, 6:01 PM
@witty-helmet-74677 just to be sure I'm understanding, you're populating a bucket with an external program then doing a force delete?
as pointed out by @white-balloon-205 (thanks!) it seems we inherit this from the upstream AWS provider: https://github.com/hashicorp/terraform-provider-aws/issues/12146
w

witty-helmet-74677

08/31/2021, 8:27 AM
Thanks @billowy-army-68599. Answer to your question is yes, bucket is populated with a external program (actually to reproduce I just synchronize a local folder into a bucket: aws sync - this is the fastest way I found) and then I destroy the cluster which will call the force_delete.
these are the args used to create the bucket: s3.NewBucket(ctx, "my-bucket-12354698", &s3.BucketArgs{ AccelerationStatus: nil, Acl: pulumi.String("private"), Arn: nil, Bucket: nil, BucketPrefix: nil, CorsRules: nil, ForceDestroy: pulumi.Bool(true), Grants: nil, HostedZoneId: nil, LifecycleRules: s3.BucketLifecycleRuleArray{&s3.BucketLifecycleRuleArgs{Enabled: pulumi.Bool(true), Expiration: &s3.BucketLifecycleRuleExpirationArgs{Days: pulumi.Int(10)}}}, Loggings: nil, ObjectLockConfiguration: nil, Policy: nil, ReplicationConfiguration: nil, RequestPayer: nil, ServerSideEncryptionConfiguration: &s3.BucketServerSideEncryptionConfigurationArgs{ Rule: &s3.BucketServerSideEncryptionConfigurationRuleArgs{ ApplyServerSideEncryptionByDefault: &s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs{ KmsMasterKeyId: pulumi.String("<key_id>"), SseAlgorithm: pulumi.String("aws:kms"), }, BucketKeyEnabled: pulumi.Bool(true), }, }, Tags: nil, TagsAll: nil, Versioning: nil, Website: nil, WebsiteDomain: nil, WebsiteEndpoint: nil, })