https://pulumi.com logo
#general
Title
# general
i

important-leather-28796

05/16/2019, 11:08 PM
I’ve got a production problem with rolling deployments. Our version of
api
failed to deploy (never ready), but
web
which
dependsOn: [api]
continued, and in fact it seems the pulumi command did not error in any way.
Sample code:
Copy code
const apiConfig = new Config('api')
export const api = new Api(
  {
    ...commonArgs,
    wildcardCertificate,
    replicas: apiConfig.getNumber('replicas'),
  },
  opts({ dependsOn: [apiCredentials, apiSecret, namespace, wildcardCertificate] }),
)

const webConfig = new Config('web')
export const web = new Web(
  {
    ...baseArgs,
    wildcardCertificate,
    replicas: webConfig.getNumber('replicas'),
  },
  opts({ dependsOn: [api] }),
)
I was expecting a failure to continue and a non-zero exit from pulumi
We essentially rolled out front-end code expecting the back end updated
What we got was an old version of
api
that continued to run because it was never successfully update and caused a bunch of errors due to the front end being out of sync
Is
dependsOn
just waiting on successfully updating the spec? If so, how can I wait on actually success of the
api
rollout?
w

white-balloon-205

05/17/2019, 3:52 PM
What are
Api
and
Web
in this example? Are they custom components? What do you use in the implementation of those? (Awsx.ecs.service?)
It used to be that depending on a component silently had no effect. I believe we changed this recently to have depending on a component implicitly depend on all its children. Cc @lemon-spoon-91807.
i

important-leather-28796

05/17/2019, 3:53 PM
My understanding is that I did not need to propagate the
dependsOn
inside the component as long as the deployment/service/ingress were all parented
I just added Web to that gist
pulumi version
"@pulumi/pulumi": "^0.17.8"
w

white-balloon-205

05/17/2019, 4:07 PM
My understanding is that I did not need to propagate the
dependsOn
inside the component as long as the deployment/service/ingress were all parented
This certainly was not true up until recently. We did some work to try to make it true, but I recall it ran into issues and had to be rolled back. I think we landed a modified version of this recently. @lemon-spoon-91807 can hopefully confirm.
l

lemon-spoon-91807

05/17/2019, 8:07 PM
Hi! So, to shed some light on how pulumi currently works (i.e. if you're using very recent versions of all the pulumi packages):
It used to be that you had to explicitly depend on any custom resources that were relevant to you (i.e. that you needed to be created, but you didn't take an explicit data dependency on).
so you'd have to list all those custom resources in your
dependsOn: ...
value
with 0.17.0 and upwards we've improved this somewhat.
specifically, if you have a component resource built out of several other components and several other custom resources, then you can now just depend on that top-level compoent resource, and it will transitively pick up (through parent/child relationships) all the leaf custom resources
this is useful for a couple of cases:
1. you have something like awsx.somemod.SomeComponent. These often wrap the underlying aws.somemod.SomeComponent. It was easy to accidentally
dependsOn
the awsx component and have that be meaningless. Now, that will do "the right thing" and cause you to actually depend on the underlying aws resource.
2. it's fairly common for a component resource to represent an aggregation of resources, all of which you want to be complete before you can move forward. These include security groups, roles, loadbalancers, and the like.
Now, you can just point at the component and it will properly wait for all of these.
3. you are resilient to components changing in the future. say a component added a new child in the future, and you needed to dependOn that. you'd previously had to know that we did that, and manually update your dependsOn clause. Now it happens automatigically by you just stating your top level component dependency
i hope that helps @important-leather-28796
i

important-leather-28796

05/17/2019, 9:46 PM
Thanks @lemon-spoon-91807
specifically, if you have a component resource built out of several other components and several other custom resources, then you can now just depend on that top-level compoent resource, and it will transitively pick up (through parent/child relationships) all the leaf custom resources
This is what did not happen. Pulumi updated the
api
spec deployment, the replica set was created but did not successfully achieve ready/live, and the
update
moved on to the rest of the components that depend on
api
. The cli returned 0. The system was stuck with the old replicaset running, the new replicaset failing, but no indication to us through pulumi.
l

lemon-spoon-91807

05/17/2019, 9:46 PM
can you clarify what you mean by "`api` spec deployment"?
what sort of resource is this?
l

lemon-spoon-91807

05/17/2019, 9:47 PM
where is teh 'replica set' you're referring to here?
thanks!
i

important-leather-28796

05/17/2019, 9:48 PM
the replica set generated by k8s from the
api
Deployment
rolling update
l

lemon-spoon-91807

05/17/2019, 9:48 PM
in this code sample, i would expect that someone depending on Api would dependOn deployment/service/ingress, since htey're all parented to Api
however it would also be necessary for the k8s lib to also be using the updated pulumi/pulumi in case those types are Components as well
so that that way their underlying custom resources would be found and depended on.
also (since i don't know the k8s package) it's necessary in that package for those Components (if they are components) to properly parent themselves to their underlying children
i

important-leather-28796

05/17/2019, 9:50 PM
so
web
dependsOn
api
, top comment
l

lemon-spoon-91807

05/17/2019, 9:50 PM
(like you have on in your code)
i

important-leather-28796

05/17/2019, 9:50 PM
are you talking my code or pulumi-kubernetes
l

lemon-spoon-91807

05/17/2019, 9:50 PM
pulumi-kubernetes
i

important-leather-28796

05/17/2019, 9:50 PM
ok
l

lemon-spoon-91807

05/17/2019, 9:51 PM
note:
i

important-leather-28796

05/17/2019, 9:51 PM
definitely did not work as expected, and didn’t fail which is very strange. We had no idea
l

lemon-spoon-91807

05/17/2019, 9:51 PM
so
web
dependsOn
api
, top comment
can you link me to the link you mean?
oh, i can totally expect yu ran into a problem, and that it was 0% your fault
i

important-leather-28796

05/17/2019, 9:52 PM
l

lemon-spoon-91807

05/17/2019, 9:56 PM
gotcha
what version of pulumi-kubernetes are you referencing?
i

important-leather-28796

05/17/2019, 9:57 PM
Copy code
"@pulumi/kubernetes": "^0.22.2",
    "@pulumi/pulumi": "^0.17.8"
l

lemon-spoon-91807

05/17/2019, 9:57 PM
thanks, looking
so i don't know if htat pulumi/k release would have the right pulumi/pulumi ref, but i would guess it would
is your repro self contained?
i.e. i can just pull down all this code and try this out?
if so, i can file bug on this and look into this on monday
if it isn't selfcontained, that def makes it a bit more annoying, but still doable 🙂
if you can give the full repro, i'd def appreciate it
i

important-leather-28796

05/17/2019, 10:00 PM
not so much. I can get a repo that is closer
l

lemon-spoon-91807

05/17/2019, 10:00 PM
it would not be hard on my end to at least see if we're collecting the full set of custom resources
if not, that's a bug on our side
if so, and it's still not working, that may be a bug elsewhere
i

important-leather-28796

05/17/2019, 10:00 PM
I’ll work on the repro and ping you
l

lemon-spoon-91807

05/17/2019, 10:01 PM
ok wait...
👍 1
need clarity on something.
the replica set generated by k8s from the
api
Deployment
rolling update
so... when i look at Deployment:
it's a CustomResource
and doesn't have any child resources of its own
if you change your code to directly depend onthe Deployment is everything ok?
if so, this is def a bug with us (and more specifically me)
so yaay for hitting the right person if so
😀 1
i

important-leather-28796

05/17/2019, 10:03 PM
didn’t try, this was a production revelation
l

lemon-spoon-91807

05/17/2019, 10:03 PM
yeah... if changing the dependson to the Deployment doesn't help, then this is an aspect that goes back to the K guys (i.e. not me)
i have to run. let me know if/when you have a repro
and can look this weekend
i

important-leather-28796

05/17/2019, 10:04 PM
ok thanks
l

lemon-spoon-91807

05/17/2019, 10:04 PM
er, next week
i

important-leather-28796

05/17/2019, 10:04 PM
it will be Monday
l

lemon-spoon-91807

05/17/2019, 10:04 PM
works for me
have a good weekend!
i

important-leather-28796

05/17/2019, 10:05 PM
you too! thanks
@lemon-spoon-91807 - I have reproduced my situation, though I am unclear if this is a pulumi bug
l

lemon-spoon-91807

05/20/2019, 7:23 PM
Thanks!
i

important-leather-28796

05/20/2019, 7:24 PM
If I use my
2.0
sample image, it exits 1 on start and pulumi behavior is as expected, api fails and web fails
If I rely on liveness/readiness, api/web deploy
l

lemon-spoon-91807

05/20/2019, 7:25 PM
so question from last week: if you manually change the dependsOn to the specific underlyin resource, does it then work?
i

important-leather-28796

05/20/2019, 7:26 PM
I’ll try
l

lemon-spoon-91807

05/20/2019, 7:31 PM
ok. so this isn't really to the 'dependsOn' work that i did 🙂
this sounds like the underlying resource is getting created as far as it's underlying 'provider' is concerned
i

important-leather-28796

05/20/2019, 7:31 PM
no, seems that dependsOn is wired up.
l

lemon-spoon-91807

05/20/2019, 7:32 PM
but it's not in a 'healthy' enough state.
i'm not a K expert
i

important-leather-28796

05/20/2019, 7:33 PM
It is either the k8 provider should not be expected to behave this way? or k8 provider has a bug when expecting liveness to indicate ready to move along and update next
l

lemon-spoon-91807

05/20/2019, 7:33 PM
let's start a conversation with Alex.
i

important-leather-28796

05/20/2019, 7:33 PM
I’m not sure it is a bug or not but I do have to solve it