https://pulumi.com logo
Join Slack
Powered by
I'm having an issue where a K8s deployment that is...
# general
o
I'm having an issue where a K8s deployment that is spun up via Helm chart consistently behaves in two ways: 1. A second replicaset is created when a deployment is updated (makes sense), that spins up a new replicaset, that starts up, and the first doesn't go away (???) 2. On refresh it then thinks the deployment is in an unhealthy state and subsequent deployments fail with:
Copy code
Diagnostics:
  kubernetes:extensions:Deployment ([redacted]):
    warning: Refreshed resource is in an unhealthy state:
    * Resource '[redacted]' was created but failed to initialize
    * Minimum number of Pods to consider the application live was not attained
Here's what the resources look like:
Copy code
$ kubectl -n ... get deployment
NAME         READY     UP-TO-DATE   AVAILABLE   AGE
[deployment] 3/3       3            3           11h
$ kubectl -n ... get rs
NAME    DESIRED   CURRENT   READY     AGE
[old]   0         0         0         11h
[new]   3         3         3         6m4s
c
there is no particular reason why the first would go away.
the GC might pick it up, but it depends.
o
Okay
c
we don’t control that, the deployment controller does.
o
Ah, right
Okay, any idea why the deployment is considered unhealthy on refresh?
c
it says that the resource failed to become live
what’s the status field of the deployment say?
o
the new RS and new pods are up, and the ReplicaSet is healthy:
Copy code
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
c
I mean, literally the 
.status
field in the deployment
o
Copy code
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-05-24T09:09:37Z
    lastUpdateTime: 2019-05-24T09:09:37Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 4
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
c
is this an 
extensions/v1beta1
deployment by chance?
o
it is
whoa, is there a bug with v1beta1 deployments
c
I’m guessing you’re not on the most recent pulumi kubernetes?
o
hm
may not be
I just updated Pulumi core but it's been a month or two since I updated k8s
c
it kind of is, the problem is that they’re not publishing that the rollout succeeded.
o
okay
c
we fixed this a few weeks ago
maybe months ago
anyway, try upgrading and see what happens.
o
okay, offhand, if I update kind to 
apps/v1
, what happens?
c
same issue.
oh
you mean in the definition?
o
I mean, will Pulumi replace the resource correctly?
yeah
c
it should work.
is my guess
o
this is what review app environments are for, YOLO ¯\_(ツ)_/¯
3 Views
Open in Slack
PreviousNext
Powered by