Hi, How should I handle a situation where Pulumi a...
# azure
a
Hi, How should I handle a situation where Pulumi accidentally deletes a secret (mistake in code), and it ends up in a soft-deleted state? After fixing the code, I want to continue using the same secret name. This means Pulumi needs to restore the secret, import it into the state, and then manage it as usual. How exactly does Pulumi handle the restoration of a soft-deleted secret? I understand that after restoring a soft-deleted secret, Pulumi can import it. Would running
pulumi up --refresh
be enough to detect and import the restored secret so that it can be managed again? In my case, the secret is used in CD pipeline Helm , so I don't want to change the secret name. After my experiments soft deleted KeyVault are handled by default, but secrets not
l
What does "soft deleted" mean? It's gone from state but still in the provider? You can import this, or rebuild the state by hand. You can't refresh: refresh gets values from the provider for resources it knows about. It doesn't find resources it doesn't know about. You can use
pulumi import
to resolve this, but it's probably easier to just add the
import
opt to the code for the secret, with the import id as specified in the secret's Pulumi docs (near the bottom, there should be a heading "Import", which tells you what the import id for that type of resource is). Once you've got the import opt in there, you can run a
pulumi up
and Pulumi will link the state with the deployed resource in the provider. Once you've got them linked, don't forget to remove the
import
opt. If you leave it in, you'll get warnings, and occasionally Pulumi might get confused and attempted to re-import that resource. Always safest to remove the import ids.
a
Thanks for the reply. In Azure Key Vault, deleted secrets enter a soft-deleted state and appear under “Manage deleted secrets.” To reuse a name, you must purge or recover the secret. Pulumi doesn’t automatically import soft-deleted secrets, which breaks IaC determinism. Your proposed recovery approach requires changing the pipeline, which is unacceptable in disaster-recovery scenarios where rollbacks must be seamless. I’m running Pulumi in a GitHub Actions and need to be a deterministic way to restore and manage secrets without renaming them When Pulumi destroys a secret, it removes it from the stack state. If a secret with the same name is recreated, Pulumi throws a conflict because the previous secret still exists in a soft-deleted state. Interestingly, Pulumi handles Key Vault restoration more gracefully—it can restore a soft-deleted Key Vault and auto-import it into state. In old Azure package it was handled. In new Azure Native not, I tried the recommended
pulumi up --refresh
, but it didn’t work as expected.
l
Assuming the Azure soft-deleted state is not represented in Pulumi, that part will have to be undone manually. You'll have to restore it to undeleted via the console. You won't have to change the pipeline for any of the steps I mentioned (not that they seem to be appropriate, but, if they were..). All manual recovery steps would happen via a dev machine.
a
We’ve implemented custom logic using the Azure SDK (NuGet package) to handle secrets in a soft-deleted state directly within our code. This solution works well, but we believe this is a common scenario that others in the community likely face too. Why does Pulumi treat Key Vaults and secrets differently when it comes to soft-delete recovery?
l
Beyond my knowledge at this point, sorry. Is it because azure-native is too "low level" and a more user-centric provider like the old azure one might have this sort of rich functionality added?
a
Yeah, no problem 😊 I'm just trying to understand why this inconsistency exists. Thanks for your response—I’ve also posted a GitHub issue about it. I was able to manually perform everything you suggested, but when we're building a fully automated IaC-based CI/CD pipeline, one-time modification steps are simply not acceptable. The process needs to be deterministic and repeatable, especially in production environments
l
A deterministic and repeatable mistake in code? 🙂 There'll always be an exception that requires you to short the green wire, or turn it off and turn it on again.. or log into the console. You have to be ready for those exceptional cases 🙂
a
The problem is when multiple people is working on the same code 🙂 And you have pulumi up before merge but someone run the main and remove you secrets
💯 1
l
Promise never to deploy changes to prod that haven't been deployed to at least two other environments previously 🙂
s
One thing that might be of interest to build in some custom logic is lifecycle hooks https://www.pulumi.com/docs/iac/concepts/options/hooks/. For example, you could set it so the secret is always completely destroyed and purged on delete
Copy code
const afterDelete = new pulumi.ResourceHook("after", async () => {
  console.log("Waiting for secret deletion to complete before purging...");
  const credential = new DefaultAzureCredential();
  const vaultUrl = `<https://elisabethtest-kv.vault.azure.net/>`;
  const client = new SecretClient(vaultUrl, credential);

  // Wait for the secret to be fully deleted before purging
  const maxRetries = 30;
  const retryDelay = 2000; // 2 seconds

  for (let i = 0; i < maxRetries; i++) {
    try {
      // Check if secret is in deleted state (not being deleted)
      await client.getDeletedSecret("elisabethtest");
      console.log("Secret is now in deleted state, proceeding with purge...");

      await client.purgeDeletedSecret("elisabethtest");
      console.log("Secret purged successfully");
      return;
    } catch (error: any) {
      if (error.code === "ObjectIsBeingDeleted") {
        console.log(`Attempt ${i + 1}: Secret still being deleted, waiting...`);
        await new Promise((resolve) => setTimeout(resolve, retryDelay));
        continue;
      } else if (error.code === "SecretNotFound") {
        console.log("Secret not found in deleted state, may already be purged");
        continue;
      } else {
        console.warn("Failed to purge secret:", error.message);
        return;
      }
    }
  }

  console.warn("Timeout waiting for secret deletion to complete");
});
// Create or import an Azure Key Vault secret
const keyVaultSecret = new azure.keyvault.Secret(
  "elisabethtest",
  {
    resourceGroupName: resourceGroup.name,
    vaultName: keyVault.name,
    secretName: "elisabethtest",
    properties: {
      value: "elisabeth-secret-value",
    },
  },
  {
    hooks: {
      afterDelete: [afterDelete],
    },
  }
);
👍 1
Though this would mean that the secret is not restorable if accidentally deleted, so a better strategy may be to use a beforeCreate hook to check if one already exists and import it if so