https://pulumi.com logo
Join Slack
Powered by
I'm running into an issue where my statefulset is ...
# kubernetes
c
I'm running into an issue where my statefulset is trying to update during the preview instead of replace when I make property changes, even though I've specified 
ReplaceOnChanges = ["*"]
Anyone have an idea for resolving this?
Copy code
Preview failed: resource "..." was not successfully created by the Kubernetes API server: StatefulSet.apps "cluster" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
My customresourceoptions:
Copy code
new CustomResourceOptions {
    Parent = this,
    Provider = k8sProvider,
    // this didn't work either
    // ReplaceOnChanges = [
    //     "spec.podManagementPolicy",
    //     "spec.revisionHistoryLimit",
    //     "spec.selector",
    //     "spec.selector.*",
    //     "spec.serviceName",
    //     "spec.volumeClaimTemplates",
    //     "spec.volumeClaimTemplates.*",
    // ],
    ReplaceOnChanges = ["*"],
    DeleteBeforeReplace = true,
    CustomTimeouts = new CustomTimeouts { 
        Create = TimeSpan.FromMinutes( 1 ), 
        Update = TimeSpan.FromMinutes( 1 )
    }
}
I've gotten this error when trying to update 
volumeClaimTemplates
before, and now I'm getting it attempting to update the 
selector
.
I'm using pulumi 3.208.0 dotnet 3.94.0 Pulumi.Kubernetes 4.24.0
is there anything I can do to try and debug an issue like this? even running pulumi with 
--verbose 4
doesn't really lay bare the inner workings of what's happening in the provider/engine
Untitled
Untitled
the full run log, if it helps any
l
What resource are you passing this CustomResourceOptions to? There's so many versions of resources and wrappers and what not in k8s-land that you may need to provide some code for diagnosis.
c
I'm working on a repro rn
👍 1
the resource is a 
StatefulSet
from 
Pulumi.Kubernetes.Apps.V1
the cluster itself is a microk8s cluster on v1.32.9 -- not sure what that translates to in kubernetes version terms 😅
l
Is this C#?
c
yes
l
I think the syntax is 
ReplaceOnChanges = { "*" }
c
to use the repro: 1. run 
pulumi up
2. uncomment the extra selector on 
Program.cs:24
and run 
pulumi up
again
I get this output in the console on the second run:
Copy code
15:55:19 /mnt/c/Users/madgv/work/bog/stateful-set-bug-repro $ pulumi up
Previewing update (dev):
     Type                               Name                        Plan        Info
     pulumi:pulumi:Stack                stateful-set-bug-repro-dev              1 error
 +-  └─ kubernetes:apps/v1:StatefulSet  my-stateful-set             replace     [diff: ~sp

Diagnostics:
  kubernetes:apps/v1:StatefulSet (my-stateful-set):
    error: Preview failed: resource "urn:pulumi:dev::stateful-set-bug-repro::kubernetes:apps/v1:StatefulSet::my-stateful-set" was not successfully created by the Kubernetes API server: StatefulSet.apps "test" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

  pulumi:pulumi:Stack (stateful-set-bug-repro-dev):
    error: preview failed

Resources:
    +-1 to replace
    3 unchanged
    2 errored
l
It's been a long time since I wrote any C#, so I'll ask: is it relevant than 
ReplaceOnChanges = ["*"]
will use collection expressions to build a 
List<string>
and assign it, but 
ReplaceOnChanges = {"*"}
will build a 
string[]
and convert/cast it? Could the Pulumi SDK be doing something clever when converting/casting?
The type of 
ReplaceOnChanges
is 
List<string>
, not 
string[]
. Not sure if that's important.
c
["*"] is syntax sugar, c# tries to infer the collection type at compile time
l
Yes, that's right. It will infer 
List<string>
.
But the refdocs use a non-inferred 
string[]
. I wondered if that was important.
c
I can switch it out to test, but I doubt it
l
Me too. But other than raising an issue in github, I can't suggest anything else to try
c
if I make a change to a non-illegal property, the stateful set does get replaced instead of updated
it's like it's respecting replaceonchanges during apply, but not preview
l
So the assertion is failing before the valid opts are taken into consideration. And this is happening during preview? Weird.
So if you run 
up --skip-preview
will it work?
c
good question, let me see
yep no problems
Copy code
$ pulumi up --skip-preview
Updating (dev):
     Type                               Name                        Status            Info
     pulumi:pulumi:Stack                stateful-set-bug-repro-dev                    
 +-  └─ kubernetes:apps/v1:StatefulSet  my-stateful-set             replaced (2s)     [dif

Resources:
    +-1 replaced
    3 unchanged

Duration: 7s
l
Ok then, I suggest updating your issue with this extra info and clarifying that preview is the location of the problem. I've upvoted it.
c
just did that
👍 1
thanks for taking a look @little-cartoon-10569 🙂
l
No probs, sorry there wasn't an immediate solution. I hope the workaround will get you through until it's fixed.
c
yeah it's been a thorn in my side for awhile, finally bugged me enough to submit a report 😆
I was partially surprised no one else has reported it because it seems pretty easy to run into if you're using statefulsets at all
I guess if you're primarily doing helm Releases you'd sidestep it...
l
It may be C# specific. C# doesn't have a native runtime, it uses the converter architecture; maybe it's happening there?
c
hm, maybe
C# does seem to be less used
I'd take a look at tracking it down myself but the pulumi architecture is intimidating lol
l
Yea, you need a primer for sure. There's good docs in the pulumi/pulumi repo, but we all have full time jobs... 😞
c
alas
good docs is a full time job by itself
2 Views
Open in Slack
PreviousNext
Powered by