It seems as though if I stop an ec2 instance and start it up again, pulumi retains the original public ip / dns even though it has changed and even after doing

pulumi refresh

is this a bug?

That does not sound expected. Could you open an issue so we can look into it?