We ran into a weird issue with the azure-native provider. We configure a custom provider and use it to create various resources in our stack like so:

const customProvider = new azure_native.Provider("customProvider", {
    clientId: process.env.CUSTOM_PROVIDER_CLIENT_ID,
    clientSecret: secret(process.env.CUSTOM_PROVIDER_CLIENT_SECRET!)
});

const roleAssignment = new azure_native.authorization.RoleAssignment("roleAssignment", {
    principalId: objectId,
    principalType: "ServicePrincipal",
    roleAssignmentName: roleName,
    roleDefinitionId: roleId,
    scope: scope
}, {
    provider: customProvider
})

The

CUSTOM_PROVIDER_CLIENT_SECRET

gets injected via an environment variable from our pipeline, for instance. This setup worked fine for some time until our client secret expired, which we only noticed when it was too late. We then rotated our secrets, including in the pipeline. However, suddenly, all operations that somehow had to refresh the

roleAssignment

resource started failing with the following error:

Invalid client secret provided

. This can be provoked when running

pulumi up --refresh

or

pulumi destroy --target <urn of the roleAssignment>

, for instance. Are the provider credentials somehow cached in the Pulumi state? How can we fix this issue? We haven't found any fix for this, except manually deleting all affected resources from the state and re-importing them.