stale-autumn-24797
06/30/2021, 7:04 PMazure-native:containerservice:AgentPool (dg-gpuspot-agentpool-cpu-staging-dev):
error: Code="PropertyChangeNotAllowed" Message="Changing property 'properties.nodeTaints' is not allowed." Target="properties.nodeTaints"
We weren’t trying to update the node taints. It seems like Pulumi knows it was a change to maxCount:
~ azure-native:containerservice:AgentPool dg-gpuspot-agentpool-cpu-staging-dev updating [diff: ~maxCount]; error: Code="PropertyChangeNotAllowed" Message="Changing property 'properties.nodeTaints' is not allowed." Target="properties.nodeTaints"
~ azure-native:containerservice:AgentPool dg-gpuspot-agentpool-cpu-staging-dev **updating failed** [diff: ~maxCount]; error: Code="PropertyChangeNotAllowed" Message="Changing property 'properties.nodeTaints' is not allowed." Target="properties.nodeTaints"
Taking a look at the stack output for this resource, I think the issue is that Pulumi is trying to remove the taints that Azure automatically adds to spot node pools. In this case we haven’t been able to get this to work even after editing the stack.
{
"urn": "urn:pulumi:spark-staging::spark-deployment::azure-native:containerservice:AgentPool::dg-gpuspot-agentpool-cpu-staging-dev",
"custom": true,
"id": "/subscriptions/a8c3fdb1-94c2-4db4-bc18-470696fa4bd4/resourcegroups/cp-staging/providers/Microsoft.ContainerService/managedClusters/dg-spark-kubernetescluster-cpu-staging-dev/agentPools/gpuspot",
"type": "azure-native:containerservice:AgentPool",
"inputs": {
"agentPoolName": "gpuspot",
"count": 1,
"enableAutoScaling": true,
"maxCount": 6,
"minCount": 0,
"mode": "User",
"nodeTaints": [
"sku=gpu:NoSchedule"
],
"resourceGroupName": "cp-staging",
"resourceName": "dg-spark-kubernetescluster-cpu-staging-dev",
"scaleSetPriority": "Spot",
"spotMaxPrice": -1,
"type": "VirtualMachineScaleSets",
"vmSize": "Standard_NC8as_T4_v3",
"vnetSubnetID": "/subscriptions/a8c3fdb1-94c2-4db4-bc18-470696fa4bd4/resourceGroups/cp-staging/providers/Microsoft.Network/virtualNetworks/dg-spark-vnet-cpu-staging-dev/subnets/default"
},
"outputs": {
"__inputs": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "redacted"
},
"count": 1,
"enableAutoScaling": true,
"enableFIPS": false,
"id": "/subscriptions/a8c3fdb1-94c2-4db4-bc18-470696fa4bd4/resourcegroups/cp-staging/providers/Microsoft.ContainerService/managedClusters/dg-spark-kubernetescluster-cpu-staging-dev/agentPools/gpuspot",
"kubeletDiskType": "OS",
"maxCount": 6,
"maxPods": 110,
"minCount": 0,
"mode": "User",
"name": "gpuspot",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.06.02",
"nodeLabels": {
"<http://kubernetes.azure.com/scalesetpriority|kubernetes.azure.com/scalesetpriority>": "spot"
},
"nodeTaints": [
"sku=gpu:NoSchedule",
"<http://kubernetes.azure.com/scalesetpriority=spot:NoSchedule|kubernetes.azure.com/scalesetpriority=spot:NoSchedule>"
],
"orchestratorVersion": "1.19.11",
"osDiskSizeGB": 128,
"osDiskType": "Ephemeral",
"osSKU": "Ubuntu",
"osType": "Linux",
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"scaleSetEvictionPolicy": "Delete",
"scaleSetPriority": "Spot",
"spotMaxPrice": -1,
"type": "VirtualMachineScaleSets",
"vmSize": "Standard_NC8as_T4_v3",
"vnetSubnetID": "/subscriptions/a8c3fdb1-94c2-4db4-bc18-470696fa4bd4/resourceGroups/cp-staging/providers/Microsoft.Network/virtualNetworks/dg-spark-vnet-cpu-staging-dev/subnets/default"
billowy-army-68599
06/30/2021, 7:07 PMgentle-diamond-70147
06/30/2021, 7:07 PMignoreChanges: ["properties.nodeTaints"]
(or the correct resource property name) to see if Pulumi will ignore those and allow the update to proceed?dazzling-alarm-77720
06/30/2021, 7:11 PMgentle-diamond-70147
06/30/2021, 7:18 PMstale-autumn-24797
06/30/2021, 7:21 PMgentle-diamond-70147
06/30/2021, 7:22 PM