Just found the root cause of the problem. It happens when a vault that contains secrets is deleted by Pulumi destroy. Next time we run Pulumi up, and populate the vault again the issue shows up. The solution around it is to purge the vault manually before running Pulumi again, or create the vault not allowing for soft deletes.