Is there some way to create a resource when running pulumi up; but then destroy it when pulumi up is finished? (e.g. to open a firewall)

The Command provider would be one way. If the firewall vendor has a provider, then that would be another way (of course).

Yeah I've been working with the command provider

If the firewall has an API, you could choose to potentially interact with their API directly from whatever language you’re using.

It's a RouterOS Firewall, so SSH is pretty ergonomic

Fair enough!

As a slightly related question; is there a way to get something more helpful than "Process failed with exit code 1" when an ssh remote command fails?

Unfortunately, no (not AFAIK).

Depending on what exactly needs to happen, it may be easier to do that externally to pulumi. We generally run pulumi from Github Actions and there's a step that comes before the pulumi run that sets up a vpn connection; if I needed to open and close a firewall hole I would probably write a little shell or python script that does that and wraps pulumi in the middle

That's probably a better idea than what I have running