https://pulumi.com logo
Powered by
o

orange-policeman-59119

05/24/2019, 5:41 PM
I'm having an issue where a K8s deployment that is spun up via Helm chart consistently behaves in two ways: 1. A second replicaset is created when a deployment is updated (makes sense), that spins up a new replicaset, that starts up, and the first doesn't go away (???) 2. On refresh it then thinks the deployment is in an unhealthy state and subsequent deployments fail with:
Copy code
Diagnostics:
  kubernetes:extensions:Deployment ([redacted]):
    warning: Refreshed resource is in an unhealthy state:
    * Resource '[redacted]' was created but failed to initialize
    * Minimum number of Pods to consider the application live was not attained
Here's what the resources look like:
Copy code
$ kubectl -n ... get deployment
NAME         READY     UP-TO-DATE   AVAILABLE   AGE
[deployment] 3/3       3            3           11h
$ kubectl -n ... get rs
NAME    DESIRED   CURRENT   READY     AGE
[old]   0         0         0         11h
[new]   3         3         3         6m4s
c

creamy-potato-29402

05/24/2019, 5:44 PM
there is no particular reason why the first would go away.
the GC might pick it up, but it depends.
o

orange-policeman-59119

05/24/2019, 5:45 PM
Okay
c

creamy-potato-29402

05/24/2019, 5:45 PM
we don’t control that, the deployment controller does.
o

orange-policeman-59119

05/24/2019, 5:45 PM
Ah, right
Okay, any idea why the deployment is considered unhealthy on refresh?
c

creamy-potato-29402

05/24/2019, 5:46 PM
it says that the resource failed to become live
what’s the status field of the deployment say?
o

orange-policeman-59119

05/24/2019, 5:46 PM
the new RS and new pods are up, and the ReplicaSet is healthy:
Copy code
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
c

creamy-potato-29402

05/24/2019, 5:47 PM
I mean, literally the 
.status
field in the deployment
o

orange-policeman-59119

05/24/2019, 5:48 PM
Copy code
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2019-05-24T09:09:37Z
    lastUpdateTime: 2019-05-24T09:09:37Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 4
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
c

creamy-potato-29402

05/24/2019, 5:48 PM
is this an 
extensions/v1beta1
deployment by chance?
o

orange-policeman-59119

05/24/2019, 5:49 PM
it is
whoa, is there a bug with v1beta1 deployments
c

creamy-potato-29402

05/24/2019, 5:49 PM
I’m guessing you’re not on the most recent pulumi kubernetes?
o

orange-policeman-59119

05/24/2019, 5:49 PM
hm
may not be
I just updated Pulumi core but it's been a month or two since I updated k8s
c

creamy-potato-29402

05/24/2019, 5:49 PM
it kind of is, the problem is that they’re not publishing that the rollout succeeded.
o

orange-policeman-59119

05/24/2019, 5:50 PM
okay
c

creamy-potato-29402

05/24/2019, 5:50 PM
we fixed this a few weeks ago
maybe months ago
anyway, try upgrading and see what happens.
o

orange-policeman-59119

05/24/2019, 5:52 PM
okay, offhand, if I update kind to 
apps/v1
, what happens?
c

creamy-potato-29402

05/24/2019, 5:53 PM
same issue.
oh
you mean in the definition?
o

orange-policeman-59119

05/24/2019, 5:53 PM
I mean, will Pulumi replace the resource correctly?
yeah
c

creamy-potato-29402

05/24/2019, 5:53 PM
it should work.
is my guess
o

orange-policeman-59119

05/24/2019, 5:53 PM
this is what review app environments are for, YOLO ¯\_(ツ)_/¯
Powered by