Currently I'm fixing it by running

pulumi export

and removing the deleted resource from the JSON