This issue should provide context:
https://github.com/pulumi/pulumi-command/issues/167
As to why it happens, it's likely an oversight.
I personally didn't know that environment variables are part of the size limit for arguments, and it looks like the same kind of issue impacts others too (such as Argo).
The fix would be to use temporary files to store the stdout/stderr, and truncate the values for backwards compatibility while users migrate.