https://pulumi.com logo
#general
Title
# general
s

square-intern-22004

11/13/2019, 12:54 AM
Hello, I have a k8s Deployment (TypeScript) I am updating via
pulumi up
in which I have left out needed env vars for my container so the Pod enters a CrashLoopBackOff, I would expect the command to eventually fail with a non-zero exit code. However, all it does is warn about the CrashLoopBackOff and then completes with 0 exit. In k8s I can see the old revision is still working as expected and the new revision as status CrashLoopBackOff. Is there something I need to specify either in the cli command or my configuration to get the desired failure exit to occur?
c

creamy-potato-29402

11/13/2019, 1:09 AM
It should definitely fail with error code 1.
does it print a message at the end with something like
error
in it?
s

square-intern-22004

11/13/2019, 1:21 AM
Copy code
$ pulumi up
Previewing update (feature):

     Type                           Name                    Plan       Info
     pulumi:pulumi:Stack            my-deployment-feature
 ~   └─ kubernetes:apps:Deployment  my-deployment          update     [diff: ~spec]

Outputs:
  ~ name: "my-deployment-y96by901" => output<string>

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (feature):

     Type                           Name                    Status      Info
     pulumi:pulumi:Stack            my-deployment-feature
 ~   └─ kubernetes:apps:Deployment  my-deployment          updated     [diff: ~spec]

Outputs:
    name: "my-deployment-y96by901"

Resources:
    ~ 1 updated
    1 unchanged

Duration: 2m57s

Permalink: <https://app.pulumi.com/><user>/my-deployment/feature/updates/4
$ echo $?
0
This is one where I got it to fail by using an image that didn’t exist which was expected.
Copy code
$ pulumi up
Previewing update (feature):

     Type                           Name                    Plan       Info
     pulumi:pulumi:Stack            my-deployment-feature             
 ~   └─ kubernetes:apps:Deployment  my-deployment          update     [diff: ~spec]
 
Outputs:
  ~ name: "my-deployment-y96by901" => output<string>

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (feature):

     Type                           Name                    Status                  Info
     pulumi:pulumi:Stack            my-deployment-feature  **failed**              1 error
 ~   └─ kubernetes:apps:Deployment  my-deployment          **updating failed**     [diff: ~spec]; 1 error

Diagnostics:
  pulumi:pulumi:Stack (my-deployment-feature):
    error: update failed
 
  kubernetes:apps:Deployment (my-deployment):
    error: Plan apply failed: 4 errors occurred:
    	* the Kubernetes API server reported that "namespace/my-deployment-y96by901" failed to fully initialize or become live: 'my-deployment-y96by901' timed out waiting to be Ready
    	* [MinimumReplicasUnavailable] Deployment does not have minimum availability.
    	* Minimum number of live Pods was not attained
    	* [Pod namespace/my-deployment-y96by901-775cbd5c9f-zwvvh]: containers with unready status: [my-deployment] -- [ImagePullBackOff] Back-off pulling image "my-deployment:pulumi-missing"
 
Outputs:
  - name: "my-deployment-y96by901"

Resources:
    1 unchanged

Duration: 10m3s

Permalink: <https://app.pulumi.com/><user>/my-deployment/feature/updates/3
$ echo $?
255
w

white-balloon-205

11/13/2019, 2:29 AM
On that first case- it appears the update is indeed succeeding. If the Deployments pods are permanently stuck crash looping, that is unexpected. Do you happen to have a repro that causes this you could share?
s

square-intern-22004

11/13/2019, 2:38 AM
Hey Luke, I specifically caused the pod to fail by not providing env vars that the node js application inside expects. So the CrashLoopBackOff is expected. What I did not expect was the
pulumi up
command to pass in that case, as i believe
kubectl rolling-update
or
kubectl apply
with a rolling update configured in the deployment with crashing pods on the new revision will eventually fail.
You could simulate by just have a Docker container that runs
exit 1
as its entrypoint and deploying it out
c

creamy-potato-29402

11/13/2019, 3:23 AM
Depends how your liveness/readiness probes are set up
if they’re set up to succeed before the pod crashes, they’ll report success, and we’ll say you succeeded.
we see the first of those outputs all the time under those circumstances.
s

square-intern-22004

11/13/2019, 5:09 AM
Ahhhhh that makes sense. I had not setup liveness/readiness probes yet. I’ll retry in the morning with those configured. Thanks for the help!
Just wanted to post back in here, adding liveness/readiness checks caused it to fail as expected. Thanks for the help!
👍 1
14 Views