https://pulumi.com logo
Title
b

bitter-carpenter-93554

10/27/2022, 9:36 PM
Is there any pulumi operator configuration that can help here to remove all stacks locks before restarting / terminating operator leader?
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2022-10-27T20:31:51Z"
  generateName: pulumi-kubernetes-operator-6677a05e-5955496654-
  labels:
    name: pulumi-kubernetes-operator
    pod-template-hash: "5955496654"
  name: pulumi-kubernetes-operator-6677a05e-5955496654-8x2mh
  namespace: pulumi
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: pulumi-kubernetes-operator-6677a05e-5955496654
    uid: bfa44684-a78d-4cab-ad1d-f9bbbb14854a
  resourceVersion: "3734233"
  uid: bbefaffb-6e24-4368-a1b0-e4ad32c5c9ad
spec:
  containers:
  - args:
    - --zap-level=error
    - --zap-time-encoding=iso8601
    command:
    - pulumi-kubernetes-operator
    env:
    - name: WATCH_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: OPERATOR_NAME
      value: pulumi-kubernetes-operator
    - name: GRACEFUL_SHUTDOWN_TIMEOUT_DURATION
      value: 5m
    - name: MAX_CONCURRENT_RECONCILES
      value: "10"
    image: pulumi/pulumi-kubernetes-operator:v1.9.0
    imagePullPolicy: Always
    name: pulumi-kubernetes-operator
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-4zgh4
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: <http://sj1010010242096.corp.adobe.com|sj1010010242096.corp.adobe.com>
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: pulumi-kubernetes-operator-c273094d
  serviceAccountName: pulumi-kubernetes-operator-c273094d
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: <http://node.kubernetes.io/not-ready|node.kubernetes.io/not-ready>
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: <http://node.kubernetes.io/unreachable|node.kubernetes.io/unreachable>
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-4zgh4
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-10-27T20:31:51Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-10-27T20:31:58Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-10-27T20:31:58Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-10-27T20:31:51Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: <containerd://13ed90f085d61c87f26a6d6793494910708d4ade60f631894aa4b405bfe7e88>f
    image: <http://docker.io/pulumi/pulumi-kubernetes-operator:v1.9.0|docker.io/pulumi/pulumi-kubernetes-operator:v1.9.0>
    imageID: <http://docker.io/pulumi/pulumi-kubernetes-operator@sha256:edb35eef05d6639181a61f09fc41bbd8d7be0a136564d5690928c9c0e2afed92|docker.io/pulumi/pulumi-kubernetes-operator@sha256:edb35eef05d6639181a61f09fc41bbd8d7be0a136564d5690928c9c0e2afed92>
    lastState: {}
    name: pulumi-kubernetes-operator
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-10-27T20:31:58Z"
  hostIP: 10.10.242.96
  phase: Running
  podIP: 10.42.0.78
  podIPs:
  - ip: 10.42.0.78
  qosClass: BestEffort
  startTime: "2022-10-27T20:31:51Z"
e

eager-football-6317

10/28/2022, 10:39 AM
Good question. The request context is passed through to the automation API (which essentially execs
pulumi up
), so in principle, it can be interrupted and I’d expect that to gracefully release locks. You already have graceful termination period of five minutes set on the deployment (I think it’s in the example config). So I think it bears some investigation. Would you mind posting an issue about this? https://github.com/pulumi/pulumi-kubernetes-operator/issues/new/choose
b

bitter-carpenter-93554

11/05/2022, 12:10 AM