https://pulumi.com logo
Title
a

ancient-eve-13947

02/07/2022, 1:13 PM
Hi, we had an incident yesterday where after a
pulumi refresh
command pulumi for the next
pulumi up
got the idea that an sql server instance had to be re-created. (Despite our Pulumi code regarding the databases not having changed at all.) As a result, all databases on said server were lost. Luckily, this was our DEV environment, so no harm done, BUT I would like to investigate why this happened, to ensure something like this can never happen on PROD. How do we go about this? I can point you guys at the relevant pulumi up and refresh builds, I assume you can see more in your logs than I do, to figure out why this happened. On a separate note, I think that even if pulumi decides a database server needs to be re-created, at the very least it should come with a HUGE warning that one needs to actively confirm. Even better would be if it copied the databases, so no data gets lost.
q

quiet-wolf-18467

02/07/2022, 2:01 PM
Hi Mark. You can mark the database resources as
protected: true
and this will block this from happening in your automation.
Your
pulumi refresh
probably found a default value on the Pulumi side that was different from the server side with your DB. The refresh/up logs should show this change and if it's irrelevant, it can be disregarded with
ignoreChanges: []
If you share some logs, I'd be happy to help you debug what happened
a

ancient-eve-13947

02/07/2022, 3:03 PM
Hi rawkode! first of all, where do I add that attribute "protected..."? second, I think I might have found out what triggers the re-creation in our case: when we go into the Azure portal and set a user as AAD sql admin on the server. (I have the same thing happening right now, I ran pulumi refresh, then pulumi preview, and I see it wants to re-create the server). re logs: here comes the portion from pulumi preview concerning the sql-server: ``` sqlserver (azure-native😒ql:Server) -- azure-native😒ql:Server (delete-replaced) [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Sql/servers/sqlserver85693a2b] [urn=urn😛ulumi:Development:☁️:azure-native:sql:Server::sqlserver] __createBeforeDelete : true administratorLogin : "deonsqladmin" administratorLoginPassword: "[secret]" administrators : { administratorType: "ActiveDirectory" login : "hajek@deon.de" principalType : "Group" sid : "576a4759-7452-407f-8988-e1795b8987eb" tenantId : "4194f5e7-c264-402f-b50a-1771cb365744" } location : "westeurope" minimalTlsVersion : "1.2" resourceGroupName : "rgDEV" serverName : "sqlserver85693a2b" blobServiceProperties (azure-native😒torage:BlobServiceProperties) ~ azure-native😒torage:BlobServiceProperties (update) [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Storage/storageAccounts/bsfontsb0c10337/blobServices/default] [urn=urn😛ulumi:Development:☁️:azure-native:storage:BlobServiceProperties::blobServiceProperties] __inputs : "[secret]" deleteRetentionPolicy : {"enabled":false} => Output<T> id : "/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Storage/storageAccounts/bsfontsb0c10337/blobServices/default" => Output<T> sku : {"name":"Standard_RAGRS","tier":"Standard"} => Output<T> type : "Microsoft.Storage/storageAccounts/blobServices" => Output<T>
(sry, I didn't get the block-code format right)
ah, in the generic resource options, like dependsOn
hmm. so I added `{protect:true}`to all our resources that are persistence related and ran
pulumi preview
again. now I see a lock-symbol for all these resources - except the sql server: it still says "-- azure-native😒ql:Server (delete-replaced)" etc.
maybe this works only on the database resources, but not the server? because they have the lock icon. I'll run a pulumi up and see what happens.
q

quiet-wolf-18467

02/07/2022, 3:26 PM
can you show me
pulumi preview --diff
?
a

ancient-eve-13947

02/07/2022, 3:27 PM
sec
running...
there will be other diffs, too, which are intentional.
q

quiet-wolf-18467

02/07/2022, 3:29 PM
You only need to share the diff for the resource that's causing you problems
a

ancient-eve-13947

02/07/2022, 3:29 PM
ah, sry
q

quiet-wolf-18467

02/07/2022, 3:30 PM
It's OK 🙂
a

ancient-eve-13947

02/07/2022, 3:30 PM
line 306
the code I'm using to define the server is:
q

quiet-wolf-18467

02/07/2022, 3:44 PM
OK
+-azure-native:sql:Server: (replace) 🔒
        [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Sql/servers/sqlserver85693a2b]
        [urn=urn:pulumi:Development::Cloud::azure-native:sql:Server::sqlserver]
        [provider=urn:pulumi:Development::Cloud::pulumi:providers:azure-native::default_1_47_0::c3446659-4061-4fee-9512-9ef38b316f93]
      - administrators: {
          - administratorType: "ActiveDirectory"
          - login            : "<mailto:hajek@deon.de|hajek@deon.de>"
          - principalType    : "Group"
          - sid              : "576a4759-7452-407f-8988-e1795b8987eb"
          - tenantId         : "4194f5e7-c264-402f-b50a-1771cb365744"
        }
a

ancient-eve-13947

02/07/2022, 3:44 PM
yes?
q

quiet-wolf-18467

02/07/2022, 3:44 PM
you can add
ignoreChanges: ["administrators"]
to stop this replace
a

ancient-eve-13947

02/07/2022, 3:44 PM
where do I add this?
q

quiet-wolf-18467

02/07/2022, 3:44 PM
next to
protect
a

ancient-eve-13947

02/07/2022, 3:45 PM
ah!
okay, that is very good to know
q

quiet-wolf-18467

02/07/2022, 3:45 PM
The reason for the change is that your declaration doesn't provide
administrators
directly
and Pulumi will see this as a change during a refresh
a

ancient-eve-13947

02/07/2022, 3:45 PM
yes, that makes sense
q

quiet-wolf-18467

02/07/2022, 3:45 PM
Why the protect doesn't work is confusing me though
a

ancient-eve-13947

02/07/2022, 3:46 PM
yes, me, too. also, I wonder what would happen if it tried to recreate the server, while the databases are protected. I'd hope the re-create would fail.
I will add the ignoreChanges now and run preview agian
q

quiet-wolf-18467

02/07/2022, 3:49 PM
Cool
Let me know if it doesn't work and I'll help debug
a

ancient-eve-13947

02/07/2022, 3:51 PM
works!
thanks a lot!
q

quiet-wolf-18467

02/07/2022, 3:51 PM
hi five
😁 1
a

ancient-eve-13947

02/07/2022, 4:02 PM
one more question: that ignoreChanges property, how does it match? simple string.contains()? just if we need this in the future for other stuff...
ah, nevermind, I found it here - often googling pulumi searchterm is easier for finding docs in pulumi than looking in the TOC ;P