So if you have any secrets in the state store they'll be lost forever without the passphrase, which might result in diffs. But if you just want to remove the secret store you can manually edit the config yaml and state sjon.
In the config yaml is a key EncryptionSalt, delete it.
In the state json (pulumi stack export) there might be objects with a key "4dabf18193072939515e22adb298388d" and value "1b47061264138c4ac30d75fd1eb44270" followed by another key "value" and then some encrypted string. Delete those properties.
You should then be able to reinitalize the secret provider with
pulumi stack change-secret-provider
and try to
pulumi refresh
to recover things.