https://pulumi.com logo
Join Slack
Powered by
Why would pulumi appear to upload a file to S3 (as...
# aws
h
Why would pulumi appear to upload a file to S3 (as part of updating a stack), and the file is 4MB on disk, but the S3 object has 0 bytes?
Using pulumi 2.3.0
It seems to occur after I've created a stack and I'm updating it with new files. I'll post some cloudtrail events when I'm able.
Background is I'm uploading a JavaScript bundle to S3. On each build, I redeploy. The name of the bundle changes each time (as does its ID). However the stack is created the first time and updated thereafter.
CloudTrail shows the bundle is uploaded the first time with 4MB transferred in. Subsequent logs (for each update to the stack) show the new bundles uploaded with 0 bytes transferred in (but on disk the bundle is still 4mb). 🤯
My deployment script (TypeScript) does look like this:
Copy code
async function go() {

}

outputs = go()
Where 
go
does all the work. 
outputs
contains a promise with stack outputs that I am able to print. However, could the promise be completing before all files upload? This has been working for months but I could have updated a node version or similar ...
w
This is likely the issue tracked in https://github.com/pulumi/pulumi/issues/3900#issuecomment-631641098. Does the workaround there help?
h
I will try that! Thank you for the pointer. When would it make sense to delete those files? After creating the S3 bucket objects?
w
I’m not yet sure what causes those files to be corrupted in the first place unfortunately. If you find you are seeing them corrupted repeatedly and/or predictably pleas do share details on the issue. Else my assumption is that manually removing them if you hit this will resolve the problem.
h
Forutnately a good indicator this has happened is I will see a '"bytesTransferredIn": 0' event in my CloudTrail logs. From that I can tell when this started happening. Unfortunately nothing changed in our build system in that time. Similarly, we've been using 2.3.0 since it came out, so not directly attributable to that. Trying logging around those 
/tmp
assets now, maybe that will tell me more. I'll report back as I find stuff.
What if a stack simultaneously deletes file 
A
and uploads file 
B
, but those files have identical content? I noticed in the logs that a file I was deleting and another I was uploading had identical hashes:
Copy code
+ aws:s3/bucketObject:BucketObjectN: (create)
        [urn=urn:pulumi:deploy-ci-ap-1578-contacts-graphql::deploy::aws:s3/bucketObject:BucketObject::static/js/main.1f4cd4af.chunk.js-upd]
        contentType : "application/javascript"
        forceDestroy: false
        key         : "static/js/main.1f4cd4af.chunk.js"
        source      : asset(file:8ced15a) { /root/console-ci/static/js/main.1f4cd4af.chunk.js }

  - aws:s3/bucketObject:BucketObject: (delete)
      [urn=urn:pulumi:deploy-ci-ap-1578-contacts-graphql::deploy::aws:s3/bucketObject:BucketObject::static/js/main.2e52d953.chunk.js-upd]
      contentType              : "application/javascript"
      etag                     : "d41d8cd98f00b204e9800998ecf8427e"
      forceDestroy             : false
      key                      : "static/js/main.2e52d953.chunk.js"
      source                   : "/tmp/pulumi-asset-8ced15aac0a318134f59b6ac5c8144920dd0f29f94d10f7f8a7ef4579bd1300f"
Notice that 
asset(file:8ced15a)
is the same prefix as the 
pulumi-asset
file. That 
pulumi-asset
file has 0 bytes in 
/tmp
as well:
Copy code
-rw------- 1 root root       0 Jun  7 02:17 /tmp/pulumi-asset-8ced15aac0a318134f59b6ac5c8144920dd0f29f94d10f7f8a7ef4579bd1300f
The file 
main.1f4cd4af.chunk.js
ends up with 0 bytes in the S3 bucket (and 
main.2e52d953.chunk.js
) is deleted. It seems that once a file is 0-bytes, it is always 0-bytes in subsequent updates as wesll.
w
Thanks for this analysis - that looks like a very promising lead on the root cause!
Added this to the issue as well.
h
:+1 Glad to. I love pulumi 🙂
Hope it isn't a false lead!
Unfortunately this bug bit me again (with pulumi 2.5.0). I added a comment to the existing issue at https://github.com/pulumi/pulumi/issues/3900#issuecomment-654463144.
w
Just left a reply - the fix is in the Azure provider - and in version 
2.9.1
. It looks like you might still be using 
2.8.0
?
h
Dang, that's right! I'll update.
5 Views
Open in Slack
PreviousNext
Powered by