one solution that came to mind is some kind of error lib that wraps the Pulumi errors (which don't include line numbers or anything useful besides the AWS API error message) in another error that contains line info, etc. But I would kind of expect Pulumi errors to have that?

That's exactly what we do, we have an internal error lib that wraps and enriches pulumi errors so we can more easily track them. There's also a verbosity flag you can add to the

up/preview

the verbosity flag didn't seem to give me better error context, i.e stacktrace/line number/resource name