a

    ancient-eve-13947

    7 months ago
    Hi, we had an incident yesterday where after a
    pulumi refresh
    command pulumi for the next
    pulumi up
    got the idea that an sql server instance had to be re-created. (Despite our Pulumi code regarding the databases not having changed at all.) As a result, all databases on said server were lost. Luckily, this was our DEV environment, so no harm done, BUT I would like to investigate why this happened, to ensure something like this can never happen on PROD. How do we go about this? I can point you guys at the relevant pulumi up and refresh builds, I assume you can see more in your logs than I do, to figure out why this happened. On a separate note, I think that even if pulumi decides a database server needs to be re-created, at the very least it should come with a HUGE warning that one needs to actively confirm. Even better would be if it copied the databases, so no data gets lost.
    q

    quiet-wolf-18467

    7 months ago
    Hi Mark. You can mark the database resources as
    protected: true
    and this will block this from happening in your automation.
    Your
    pulumi refresh
    probably found a default value on the Pulumi side that was different from the server side with your DB. The refresh/up logs should show this change and if it's irrelevant, it can be disregarded with
    ignoreChanges: []
    If you share some logs, I'd be happy to help you debug what happened
    a

    ancient-eve-13947

    7 months ago
    Hi rawkode! first of all, where do I add that attribute "protected..."? second, I think I might have found out what triggers the re-creation in our case: when we go into the Azure portal and set a user as AAD sql admin on the server. (I have the same thing happening right now, I ran pulumi refresh, then pulumi preview, and I see it wants to re-create the server). re logs: here comes the portion from pulumi preview concerning the sql-server: ``` sqlserver (azure-nativeπŸ˜’ql:Server) -- azure-nativeπŸ˜’ql:Server (delete-replaced) [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Sql/servers/sqlserver85693a2b] [urn=urnπŸ˜›ulumi:Development:☁️:azure-native:sql:Server:πŸ˜’qlserver] __createBeforeDelete : true administratorLogin : "deonsqladmin" administratorLoginPassword: "[secret]" administrators : { administratorType: "ActiveDirectory" login : "hajek@deon.de" principalType : "Group" sid : "576a4759-7452-407f-8988-e1795b8987eb" tenantId : "4194f5e7-c264-402f-b50a-1771cb365744" } location : "westeurope" minimalTlsVersion : "1.2" resourceGroupName : "rgDEV" serverName : "sqlserver85693a2b" blobServiceProperties (azure-nativeπŸ˜’torage:BlobServiceProperties) ~ azure-nativeπŸ˜’torage:BlobServiceProperties (update) [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Storage/storageAccounts/bsfontsb0c10337/blobServices/default] [urn=urnπŸ˜›ulumi:Development:☁️:azure-native:storage:BlobServiceProperties::blobServiceProperties] __inputs : "[secret]" deleteRetentionPolicy : {"enabled":false} => Output<T> id : "/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Storage/storageAccounts/bsfontsb0c10337/blobServices/default" => Output<T> sku : {"name":"Standard_RAGRS","tier":"Standard"} => Output<T> type : "Microsoft.Storage/storageAccounts/blobServices" => Output<T>
    (sry, I didn't get the block-code format right)
    ah, in the generic resource options, like dependsOn
    hmm. so I added {protect:true}to all our resources that are persistence related and ran
    pulumi preview
    again. now I see a lock-symbol for all these resources - except the sql server: it still says "-- azure-nativeπŸ˜’ql:Server (delete-replaced)" etc.
    maybe this works only on the database resources, but not the server? because they have the lock icon. I'll run a pulumi up and see what happens.
    q

    quiet-wolf-18467

    7 months ago
    can you show me
    pulumi preview --diff
    ?
    a

    ancient-eve-13947

    7 months ago
    sec
    running...
    there will be other diffs, too, which are intentional.
    q

    quiet-wolf-18467

    7 months ago
    You only need to share the diff for the resource that's causing you problems
    a

    ancient-eve-13947

    7 months ago
    ah, sry
    q

    quiet-wolf-18467

    7 months ago
    It's OK πŸ™‚
    a

    ancient-eve-13947

    7 months ago
    line 306
    the code I'm using to define the server is:
    q

    quiet-wolf-18467

    7 months ago
    OK
    +-azure-native:sql:Server: (replace) ≑ƒâÆ
            [id=/subscriptions/25110286-d288-4e46-851b-4e2bc880672f/resourceGroups/rgDEV/providers/Microsoft.Sql/servers/sqlserver85693a2b]
            [urn=urn:pulumi:Development::Cloud::azure-native:sql:Server::sqlserver]
            [provider=urn:pulumi:Development::Cloud::pulumi:providers:azure-native::default_1_47_0::c3446659-4061-4fee-9512-9ef38b316f93]
          - administrators: {
              - administratorType: "ActiveDirectory"
              - login            : "<mailto:hajek@deon.de|hajek@deon.de>"
              - principalType    : "Group"
              - sid              : "576a4759-7452-407f-8988-e1795b8987eb"
              - tenantId         : "4194f5e7-c264-402f-b50a-1771cb365744"
            }
    a

    ancient-eve-13947

    7 months ago
    yes?
    q

    quiet-wolf-18467

    7 months ago
    you can add
    ignoreChanges: ["administrators"]
    to stop this replace
    a

    ancient-eve-13947

    7 months ago
    where do I add this?
    q

    quiet-wolf-18467

    7 months ago
    next to
    protect
    a

    ancient-eve-13947

    7 months ago
    ah!
    okay, that is very good to know
    q

    quiet-wolf-18467

    7 months ago
    The reason for the change is that your declaration doesn't provide
    administrators
    directly
    and Pulumi will see this as a change during a refresh
    a

    ancient-eve-13947

    7 months ago
    yes, that makes sense
    q

    quiet-wolf-18467

    7 months ago
    Why the protect doesn't work is confusing me though
    a

    ancient-eve-13947

    7 months ago
    yes, me, too. also, I wonder what would happen if it tried to recreate the server, while the databases are protected. I'd hope the re-create would fail.
    I will add the ignoreChanges now and run preview agian
    q

    quiet-wolf-18467

    7 months ago
    Cool
    Let me know if it doesn't work and I'll help debug
    a

    ancient-eve-13947

    7 months ago
    works!
    thanks a lot!
    q

    quiet-wolf-18467

    7 months ago
    hi five
    a

    ancient-eve-13947

    7 months ago
    one more question: that ignoreChanges property, how does it match? simple string.contains()? just if we need this in the future for other stuff...
    ah, nevermind, I found it here - often googling pulumi searchterm is easier for finding docs in pulumi than looking in the TOC ;P