https://pulumi.com logo
#aws
Title
# aws
h

hundreds-musician-51496

06/06/2020, 5:56 PM
Why would pulumi appear to upload a file to S3 (as part of updating a stack), and the file is 4MB on disk, but the S3 object has 0 bytes?
Using pulumi 2.3.0
It seems to occur after I've created a stack and I'm updating it with new files. I'll post some cloudtrail events when I'm able.
Background is I'm uploading a JavaScript bundle to S3. On each build, I redeploy. The name of the bundle changes each time (as does its ID). However the stack is created the first time and updated thereafter.
CloudTrail shows the bundle is uploaded the first time with 4MB transferred in. Subsequent logs (for each update to the stack) show the new bundles uploaded with 0 bytes transferred in (but on disk the bundle is still 4mb). 🤯
My deployment script (TypeScript) does look like this:
Copy code
async function go() {

}

outputs = go()
Where
go
does all the work.
outputs
contains a promise with stack outputs that I am able to print. However, could the promise be completing before all files upload? This has been working for months but I could have updated a node version or similar ...
w

white-balloon-205

06/06/2020, 11:23 PM
This is likely the issue tracked in https://github.com/pulumi/pulumi/issues/3900#issuecomment-631641098. Does the workaround there help?
h

hundreds-musician-51496

06/06/2020, 11:29 PM
I will try that! Thank you for the pointer. When would it make sense to delete those files? After creating the S3 bucket objects?
w

white-balloon-205

06/07/2020, 12:10 AM
I’m not yet sure what causes those files to be corrupted in the first place unfortunately. If you find you are seeing them corrupted repeatedly and/or predictably pleas do share details on the issue. Else my assumption is that manually removing them if you hit this will resolve the problem.
h

hundreds-musician-51496

06/07/2020, 1:40 AM
Forutnately a good indicator this has happened is I will see a '"bytesTransferredIn": 0' event in my CloudTrail logs. From that I can tell when this started happening. Unfortunately nothing changed in our build system in that time. Similarly, we've been using 2.3.0 since it came out, so not directly attributable to that. Trying logging around those
/tmp
assets now, maybe that will tell me more. I'll report back as I find stuff.
What if a stack simultaneously deletes file
A
and uploads file
B
, but those files have identical content? I noticed in the logs that a file I was deleting and another I was uploading had identical hashes:
Copy code
+ aws:s3/bucketObject:BucketObjectN: (create)
        [urn=urn:pulumi:deploy-ci-ap-1578-contacts-graphql::deploy::aws:s3/bucketObject:BucketObject::static/js/main.1f4cd4af.chunk.js-upd]
        contentType : "application/javascript"
        forceDestroy: false
        key         : "static/js/main.1f4cd4af.chunk.js"
        source      : asset(file:8ced15a) { /root/console-ci/static/js/main.1f4cd4af.chunk.js }

  - aws:s3/bucketObject:BucketObject: (delete)
      [urn=urn:pulumi:deploy-ci-ap-1578-contacts-graphql::deploy::aws:s3/bucketObject:BucketObject::static/js/main.2e52d953.chunk.js-upd]
      contentType              : "application/javascript"
      etag                     : "d41d8cd98f00b204e9800998ecf8427e"
      forceDestroy             : false
      key                      : "static/js/main.2e52d953.chunk.js"
      source                   : "/tmp/pulumi-asset-8ced15aac0a318134f59b6ac5c8144920dd0f29f94d10f7f8a7ef4579bd1300f"
Notice that
asset(file:8ced15a)
is the same prefix as the
pulumi-asset
file. That
pulumi-asset
file has 0 bytes in
/tmp
as well:
Copy code
-rw------- 1 root root       0 Jun  7 02:17 /tmp/pulumi-asset-8ced15aac0a318134f59b6ac5c8144920dd0f29f94d10f7f8a7ef4579bd1300f
The file
main.1f4cd4af.chunk.js
ends up with 0 bytes in the S3 bucket (and
main.2e52d953.chunk.js
) is deleted. It seems that once a file is 0-bytes, it is always 0-bytes in subsequent updates as wesll.
w

white-balloon-205

06/08/2020, 6:24 PM
Thanks for this analysis - that looks like a very promising lead on the root cause!
Added this to the issue as well.
h

hundreds-musician-51496

06/08/2020, 6:28 PM
:+1 Glad to. I love pulumi 🙂
Hope it isn't a false lead!
Unfortunately this bug bit me again (with pulumi 2.5.0). I added a comment to the existing issue at https://github.com/pulumi/pulumi/issues/3900#issuecomment-654463144.
w

white-balloon-205

07/06/2020, 9:23 PM
Just left a reply - the fix is in the Azure provider - and in version
2.9.1
. It looks like you might still be using
2.8.0
?
h

hundreds-musician-51496

07/06/2020, 9:25 PM
Dang, that's right! I'll update.
3 Views