https://pulumi.com logo
Join Slack
Powered by
Hello, I have a k8s Deployment (TypeScript) I am ...
# general
s
Hello, I have a k8s Deployment (TypeScript) I am updating via 
pulumi up
in which I have left out needed env vars for my container so the Pod enters a CrashLoopBackOff, I would expect the command to eventually fail with a non-zero exit code. However, all it does is warn about the CrashLoopBackOff and then completes with 0 exit. In k8s I can see the old revision is still working as expected and the new revision as status CrashLoopBackOff. Is there something I need to specify either in the cli command or my configuration to get the desired failure exit to occur?
c
It should definitely fail with error code 1.
does it print a message at the end with something like 
error
in it?
s
Copy code
$ pulumi up
Previewing update (feature):

     Type                           Name                    Plan       Info
     pulumi:pulumi:Stack            my-deployment-feature
 ~   └─ kubernetes:apps:Deployment  my-deployment          update     [diff: ~spec]

Outputs:
  ~ name: "my-deployment-y96by901" => output<string>

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (feature):

     Type                           Name                    Status      Info
     pulumi:pulumi:Stack            my-deployment-feature
 ~   └─ kubernetes:apps:Deployment  my-deployment          updated     [diff: ~spec]

Outputs:
    name: "my-deployment-y96by901"

Resources:
    ~ 1 updated
    1 unchanged

Duration: 2m57s

Permalink: <https://app.pulumi.com/><user>/my-deployment/feature/updates/4
$ echo $?
0
This is one where I got it to fail by using an image that didn’t exist which was expected.
Copy code
$ pulumi up
Previewing update (feature):

     Type                           Name                    Plan       Info
     pulumi:pulumi:Stack            my-deployment-feature             
 ~   └─ kubernetes:apps:Deployment  my-deployment          update     [diff: ~spec]
 
Outputs:
  ~ name: "my-deployment-y96by901" => output<string>

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (feature):

     Type                           Name                    Status                  Info
     pulumi:pulumi:Stack            my-deployment-feature  **failed**              1 error
 ~   └─ kubernetes:apps:Deployment  my-deployment          **updating failed**     [diff: ~spec]; 1 error

Diagnostics:
  pulumi:pulumi:Stack (my-deployment-feature):
    error: update failed
 
  kubernetes:apps:Deployment (my-deployment):
    error: Plan apply failed: 4 errors occurred:
    	* the Kubernetes API server reported that "namespace/my-deployment-y96by901" failed to fully initialize or become live: 'my-deployment-y96by901' timed out waiting to be Ready
    	* [MinimumReplicasUnavailable] Deployment does not have minimum availability.
    	* Minimum number of live Pods was not attained
    	* [Pod namespace/my-deployment-y96by901-775cbd5c9f-zwvvh]: containers with unready status: [my-deployment] -- [ImagePullBackOff] Back-off pulling image "my-deployment:pulumi-missing"
 
Outputs:
  - name: "my-deployment-y96by901"

Resources:
    1 unchanged

Duration: 10m3s

Permalink: <https://app.pulumi.com/><user>/my-deployment/feature/updates/3
$ echo $?
255
w
On that first case- it appears the update is indeed succeeding. If the Deployments pods are permanently stuck crash looping, that is unexpected. Do you happen to have a repro that causes this you could share?
s
Hey Luke, I specifically caused the pod to fail by not providing env vars that the node js application inside expects. So the CrashLoopBackOff is expected. What I did not expect was the 
pulumi up
command to pass in that case, as i believe 
kubectl rolling-update
or 
kubectl apply
with a rolling update configured in the deployment with crashing pods on the new revision will eventually fail.
You could simulate by just have a Docker container that runs 
exit 1
as its entrypoint and deploying it out
c
Depends how your liveness/readiness probes are set up
if they’re set up to succeed before the pod crashes, they’ll report success, and we’ll say you succeeded.
we see the first of those outputs all the time under those circumstances.
s
Ahhhhh that makes sense. I had not setup liveness/readiness probes yet. I’ll retry in the morning with those configured. Thanks for the help!
Just wanted to post back in here, adding liveness/readiness checks caused it to fail as expected. Thanks for the help!
👍 1
60 Views
Open in Slack
PreviousNext
Powered by