https://pulumi.com logo
Join Slack
Powered by
also, quick Q: ```const syncedBucketFolder = new s...
# getting-started
m
also, quick Q:
Copy code
const syncedBucketFolder = new syncedFolder.S3BucketFolder(`pmb-mainapp-asset-folder-${getStack()}`, {
    path: "./../../app/assets",
    bucketName: assetsBucket.id,
    acl: aws.s3.CannedAcl.PublicRead,
    managedObjects: false,
}, { dependsOn: bucketAcl });
This code: (a) didn't upload all the files in that directory, and (b) is showing there are no changes when i change the files in that folder. Is there some kind of obvious config or issue I'm missing herE?
m
(b) sounds like it'd be a bug. For (a) though, did some files get uploaded, just not all?
m
yup
let me grab an example
image.png
could this be due to some kind of file system permission thing? I made sure that all the files are accessible as read to the pulumi process
and more importantly how do i force refresh this thing 🫠 on every deployment ideally
docs say this pulumi construct just issues a command to the underlying AWS CLI
package.json versions:
Copy code
"@pulumi/aws": "^6.2.0",
        "@pulumi/awsx": "^1.0.5",
        "@pulumi/docker": "^4.5.1",
        "@pulumi/eks": "^1.0.3",
        "@pulumi/pulumi": "^3.84.0",
        "@pulumi/random": "^4.15.1",
        "@pulumi/synced-folder": "^0.11.1"
lockfile seems latest with remote deps
upgraded aws cli from:
Copy code
â•°$ aws --version                                                                                                                                  
aws-cli/2.14.5 Python/3.11.6 Linux/5.15.133.1-microsoft-standard-WSL2 source/x86_64.ubuntu.22 prompt/off
to:
Copy code
â•°$ aws --version
aws-cli/2.15.17 Python/3.11.7 Linux/5.15.133.1-microsoft-standard-WSL2 source/x86_64.ubuntu.22 prompt/off
but no dice
hm, includeHiddenFiles: true - no dice also; digging through docs/source to maybe RCA
trying to workaround with a direct CLI sync with:
Copy code
const writeFile = new command.local.Command("writeFile-command", {
    create: pulumi.interpolate`aws s3 sync ./../../app/assets s3://${assetsBucket.bucket} --acl ${aws.s3.CannedAcl.PublicRead}`,
    // Ensure that the directory is created before writing the file
}, { dependsOn: bucketAcl });
I get the following error
Copy code
Diagnostics:
  command:local:Command (writeFile-command):
    error: fork/exec /bin/sh: argument list too long: running "aws s3 sync ./../../app/assets s3://[REDACTED] v --acl public-read":
..but when i literally copy/paste that into terminal, it works. any ideas?
ok... what 😭
Copy code
command:local:Command (writeFile-command):
    error: fork/exec /bin/sh: argument list too long: running "./sync.sh":
.................. commandlocalCommand (writeFile-command): error: fork/exec /bin/sh: argument list too long: running "echo abc":
what kind of environment does this run in? I've tried: • xargs • ulimit • MAX_ARGS • for loop no bypass that should normally work in any kind of shell works
on a deadline to deliver a customer demo tonight...
i can copy/paste any command that gets this error and it runs a-ok in: • bash • xterm • iTerm • zsh • shell
ok, i had to: • pulumi state delete the custom command run it with:
Copy code
const writeFile = new command.local.Command("writeFile-command", {
    create: assetsBucket.bucket.apply(name => {
        return `aws s3 sync ./../../app/assets s3://${name} --acl ${aws.s3.CannedAcl.PublicRead}`
    }),
    environment: {
        'AWS_PAGER': ''
    }
    // Ensure that the directory is created before writing the file
}, { dependsOn: bucketAcl });
AWS_PAGER silent
for some reason the "outputs" of this command in "pulumi stack export"
Copy code
{
                "urn": "urn:pulumi:dev::pmb-main-assets::command:local:Command::writeFile-command",
                "custom": true,
                "id": "writeFile-command2cfe44ee",
                "type": "command:local:Command",
                "inputs": {
                    "create": "aws s3 sync \"./../../app/assets\" <s3://pmb-mainapp-asset-s3-dev/>"
                },
                "outputs": {
                    "create": "aws s3 sync \"./../../app/assets\" <s3://pmb-mainapp-asset-s3-dev/>",
                    "stderr": "",
                    "stdout":
^ that stdout was an ungodly length. the entire output of the AWS CLI command that synced 1000+ files.
stack was blown and wouldnt update any resources until i did that
m
That's interesting, about AWS_PAGER. Is that set to empty in your shell env also?
m
nah, my shell parsed the whole 1000+ long file list
i can understand why this would go down like this but its definitely hard to understand/trace behavior
m
I wonder if the update would have succeeded normally if you'd set that before running 
up
.
m
the AWS_PAGER?
m
Yeah
m
let me try to delete the bucket / delete state + refresh and test
m
The process picks up your ambient AWS creds and other known AWS env vars, so I suspect it'd pick up AWS_PAGER also, but I also haven't seen this before, so it's just a hunch.
m
yeah it definitely did that
seems the difficulty here is that any command that produces sufficiently long output would put the stack into an invalid state
m
That'd explain the "partial upload" bit at leatst (I think).
Yeah that does seem worth an issue for sure.
m
eee.... now that i set the AWS_PAGER to blank it seems to pre-run it and think it doesn't need updating 😂 working on some kinda workaround to always force the custom resource to run
m
Does it though? Like did anything actually change?
m
the world just got stranger.
Copy code
const writeFile = new command.local.Command("writeFile-command", {
    create: assetsBucket.bucket.apply(name => {
        return `aws s3 sync ./../../app/assets s3://${name} --acl ${aws.s3.CannedAcl.PublicRead}`
    }),
    environment: {
        'AWS_PAGER': ''
    }
    // Ensure that the directory is created before writing the file
}, { dependsOn: bucketAcl });
with that, the command thinks its updated and even pulumi refresh after i nuked the bucket contents think its up to date
guessing i need to echo some kind of variable timestamp to force it into updating
m
This is your custom code though, not the Synced Folder component, is that right?
m
okay so this is the final version that works.... 😂
Copy code
const writeFile = new command.local.Command("writeFile-command", {
    create: assetsBucket.bucket.apply(name => {
        return `aws s3 sync ./../../app/assets s3://${name} --acl ${aws.s3.CannedAcl.PublicRead} && date +%Y%m%d%H%M%S`
    }),
    environment: {
        'AWS_PAGER': ''
    }
    // Ensure that the directory is created before writing the file
}, { dependsOn: bucketAcl });
tf... even with the date output it hasnt changed.
math dot random here we come
m
If you want to "always run" something, there's another command you should use -- https://www.pulumi.com/registry/packages/command/api-docs/local/run/
(or could use) However I'm still wondering whether setting AWS_PAGER="" (I believe that's right) on your shell and then using Synced Folder alone actually does what you need.
m
the command.local.Command seems to think it is up to date when i set blank
probably readily reproducible
if i dont set AWS_PAGER to empty it blows the stack and i have to manually state delete
most of the CICD envs and runners will understandably have it set to not blank (and I need that)
m
Interesting -- I thought that feature was mainly for interactive use.
m
normally I have the pager set to jq
some AWS commands we have produce 100,000+ lines of output so I've luckily known that setting it to blank silences
once the long output gets ran once and its saved in stack state, there seems to be no way out other than to manually delete
thats odd....
Copy code
command.local.runOutput({ 
    command: pulumi.interpolate`aws s3 sync ./../../app/assets s3://${assetsBucket.bucket} --acl ${aws.s3.CannedAcl.PublicRead}` 
});
doesn't re-run if the command eval is the same.
is that intended?
oh weird. I ran that above, then deleted everything in the bucket, then re-ran pulumi up. The command ran and re-uploaded to the bucket but the stack output was:
Copy code
Updating (REDACTED)

View in Browser (Ctrl+O): <https://app.pulumi.com/REDACTED>

     Type                 Name                 Status     
     pulumi:pulumi:Stack  pmb-main-assets-dev             

Resources:
    4 unchanged

Duration: 16s
i can live with that 🙂
thank you for the heads up to https://www.pulumi.com/registry/packages/command/api-docs/local/run/, in a time crunch and it really helped out!
m
You bet! Glad that helped.
2 Views
Open in Slack
PreviousNext
Powered by