Hi, we had an incident yesterday where after a `p...
# general
a
Hi, we had an incident yesterday where after a
pulumi refresh
command pulumi for the next
pulumi up
got the idea that an sql server instance had to be re-created. (Despite our Pulumi code regarding the databases not having changed at all.) As a result, all databases on said server were lost. Luckily, this was our DEV environment, so no harm done, BUT I would like to investigate why this happened, to ensure something like this can never happen on PROD. How do we go about this? I can point you guys at the relevant pulumi up and refresh builds, I assume you can see more in your logs than I do, to figure out why this happened. On a separate note, I think that even if pulumi decides a database server needs to be re-created, at the very least it should come with a HUGE warning that one needs to actively confirm. Even better would be if it copied the databases, so no data gets lost.
q
Hi Mark. You can mark the database resources as
protected: true
and this will block this from happening in your automation.
Your
pulumi refresh
probably found a default value on the Pulumi side that was different from the server side with your DB. The refresh/up logs should show this change and if it's irrelevant, it can be disregarded with
ignoreChanges: []
If you share some logs, I'd be happy to help you debug what happened
a
Hi rawkode! first of all, where do I add that attribute "protected..."? second, I think I might have found out what triggers the re-creation in our case: when we go into the Azure portal and set a user as AAD sql admin on the server. (I have the same thing happening right now, I ran pulumi refresh, then pulumi preview, and I see it wants to re-create the server). re logs: here comes the portion from pulumi preview concerning the sql-server: ``` sqlserver (azure-nativesqlServer) -- azure-nativesqlServer (delete-replaced) [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Sql/servers/sqlserver85693a2b] [urn=urnpulumiDevelopment:☁️azure nativesqlServer:sqlserver] __createBeforeDelete : true administratorLogin : "deonsqladmin" administratorLoginPassword: "[secret]" administrators : { administratorType: "ActiveDirectory" login : "hajek@deon.de" principalType : "Group" sid : "576a4759-7452-407f-8988-e1795b8987eb" tenantId : "4194f5e7-c264-402f-b50a-1771cb365744" } location : "westeurope" minimalTlsVersion : "1.2" resourceGroupName : "rgDEV" serverName : "sqlserver85693a2b" blobServiceProperties (azure-nativestorageBlobServiceProperties) ~ azure-nativestorageBlobServiceProperties (update) [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Storage/storageAccounts/bsfontsb0c10337/blobServices/default] [urn=urnpulumiDevelopment:☁️azure nativestorageBlobServiceProperties:blobServiceProperties] __inputs : "[secret]" deleteRetentionPolicy : {"enabled":false} => Output<T> id : "/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Storage/storageAccounts/bsfontsb0c10337/blobServices/default" => Output<T> sku : {"name":"Standard_RAGRS","tier":"Standard"} => Output<T> type : "Microsoft.Storage/storageAccounts/blobServices" => Output<T>
(sry, I didn't get the block-code format right)
ah, in the generic resource options, like dependsOn
hmm. so I added `{protect:true}`to all our resources that are persistence related and ran
pulumi preview
again. now I see a lock-symbol for all these resources - except the sql server: it still says "-- azure-nativesqlServer (delete-replaced)" etc.
maybe this works only on the database resources, but not the server? because they have the lock icon. I'll run a pulumi up and see what happens.
q
can you show me
pulumi preview --diff
?
a
sec
running...
there will be other diffs, too, which are intentional.
q
You only need to share the diff for the resource that's causing you problems
a
ah, sry
q
It's OK 🙂
a
line 306
the code I'm using to define the server is:
q
OK
Copy code
+-azure-native:sql:Server: (replace) 🔒
        [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Sql/servers/sqlserver85693a2b]
        [urn=urn:pulumi:Development::Cloud::azure-native:sql:Server::sqlserver]
        [provider=urn:pulumi:Development::Cloud::pulumi:providers:azure-native::default_1_47_0::c3446659-4061-4fee-9512-9ef38b316f93]
      - administrators: {
          - administratorType: "ActiveDirectory"
          - login            : "<mailto:hajek@deon.de|hajek@deon.de>"
          - principalType    : "Group"
          - sid              : "576a4759-7452-407f-8988-e1795b8987eb"
          - tenantId         : "4194f5e7-c264-402f-b50a-1771cb365744"
        }
a
yes?
q
you can add
ignoreChanges: ["administrators"]
to stop this replace
a
where do I add this?
q
next to
protect
a
ah!
okay, that is very good to know
q
The reason for the change is that your declaration doesn't provide
administrators
directly
and Pulumi will see this as a change during a refresh
a
yes, that makes sense
q
Why the protect doesn't work is confusing me though
a
yes, me, too. also, I wonder what would happen if it tried to recreate the server, while the databases are protected. I'd hope the re-create would fail.
I will add the ignoreChanges now and run preview agian
q
Cool
Let me know if it doesn't work and I'll help debug
a
works!
thanks a lot!
q
hi five
😁 1
a
one more question: that ignoreChanges property, how does it match? simple string.contains()? just if we need this in the future for other stuff...
ah, nevermind, I found it here - often googling pulumi searchterm is easier for finding docs in pulumi than looking in the TOC ;P