https://pulumi.com logo
#general
Title
# general
b

bitter-dentist-28132

09/11/2019, 7:06 PM
i just can't seem to get the nginx ingress helm chart installed any more. i always end up with
Copy code
error: Plan apply failed: 2 errors occurred:
	* 'nginxingresscontroller-nginx-ingress-controller' timed out waiting to be Ready
	* Minimum number of live Pods was not attained


error: Plan apply failed: 2 errors occurred:
	* 'nginxingresscontroller-nginx-ingress-default-backend' timed out waiting to be Ready
	* Minimum number of live Pods was not attained
b

broad-dog-22463

09/11/2019, 7:08 PM
Has this just started happening or has it never worked for you?
b

bitter-dentist-28132

09/11/2019, 7:11 PM
i'm not sure updating the chart seamlessly has ever worked
b

broad-dog-22463

09/11/2019, 7:11 PM
Ok
Has this been for a long time or you’ve only started recently using it?
b

bitter-dentist-28132

09/11/2019, 7:12 PM
i've only used pulumi for the past month or so.
i first ran into the problem where the helm chart sucks so i had to add values for it to not break on upgrade. now it doesn't really break, but pulumi always thinks it doesn't succeed, but i'm not sure why. nothing seems to update besides the chart label
i should probably just port all the resources to pulumi because helm is terrible, but i'd like to avoid that because i do have other things to do. but given how often it's breaking deployments.... maybe i should just bite the bullet and do it 😕
or pin it, and just know that any time it upgrades it'll break deployments.
probably worth pointing out, if i wipe out the cluster and recreate everything, the new version deploys correctly
b

broad-dog-22463

09/11/2019, 7:16 PM
Ok - that’s interesting to know
I’m asking in the team atm
b

bitter-dentist-28132

09/11/2019, 7:26 PM
based on the diff output, the only field that's changing is [deployment].metadata.labels.chart. looks like what changed is using a value if given, but doesn't affect the rendering for what i'm doing.
c

cool-egg-852

09/11/2019, 7:31 PM
We used to not use helm. Now we make heavy use of it. It’s a lot easier for third party applications. We only use native pulumi resources for anything not in the chart, or for our own applications. Having gone down the road of porting everything to pulumi previously, I warn you not to.
Have you looked at what the logs are showing for the pods? Or does it never update/create the Deployment?
b

bitter-dentist-28132

09/11/2019, 7:33 PM
@cool-egg-852 i appreciate the input. do you use nginx for your ingress controller or something else? it seems the nginx-ingress chart and pulumi don't get along.
i have. everything is a-ok.
my deployments deploy correctly
c

cool-egg-852

09/11/2019, 7:34 PM
We do not use nginx, we use istio ourselves. But having used helm with pulumi many times, I can’t say I’ve experienced any issues like this. Issues yes, but due to the charts themselves (not terribly hard to fix)
Can you post the full output (not the diff) from your
pulumi up
?
b

bitter-dentist-28132

09/11/2019, 7:37 PM
i mean, issues with charts are really issues with helm, wouldn't you say? the design issues of helm translates to issues with charts. poorly-made charts go hand-in-hand with poorly-designed helm.
i can
i run a
pulumi up --diff -y
, do you want that full output?
or just the
pulumi up --skip-preview
output?
c

cool-egg-852

09/11/2019, 7:39 PM
I can’t speak to helm long ago, but these days, I would say it’s just on the charts themselves. The only real issues I have are either badly designed charts, or charts that are missing the
namespace
because of the Tiller design. This will all be fixed though as Helm 3 doesn’t have tiller. Charts will have to add in
namespace
Just preview actually, not up
b

bitter-dentist-28132

09/11/2019, 7:43 PM
yeah, i don't know how it took until version 3 to create a sane design. being open source, you'd think the k8s community would have realized how crazy it was and made a better alternative. i'm hoping the saner design of v3 will influence the use of helm, but i'm not gonna hold my breath. even the charts in stable aren't using explicit namespaces. anyway, i'll stop with the ranting.
c

cool-egg-852

09/11/2019, 7:44 PM
Ok, and what is the diff in the metadata? I’m wondering if you are running into the same issue I have in the passed. It wouldn’t be helm related if it’s the case.
b

bitter-dentist-28132

09/11/2019, 7:45 PM
i think it's probably something pulumi-related, since the chart/resource does update:
Copy code
kubectl get deployment nginxingresscontroller-nginx-ingress-controller -o yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  ...
  labels:
    app: nginx-ingress
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: pulumi
    chart: nginx-ingress-1.20.0
but for whatever reason, pulumi thinks the deployment doesn't succeed (it does)
c

cool-egg-852

09/11/2019, 7:48 PM
I’ve experienced something similar in the past, though I don’t recall it being helm related. Have you tried
pulumi refresh
?
b

bitter-dentist-28132

09/11/2019, 7:50 PM
for past breakages from the same issue, yes, but let me try again and see what happens
well, even after the
pulumi refresh
, the
pulumi stack export
resource is still marked as unhealthy
now when i run
pulumi preview
, it wants to update the two deployments, but there's no diff.
i guess i will just file an issue against pulumi-kubernetes
c

cool-egg-852

09/11/2019, 7:57 PM
I had this before. I don’t recall how I fixed it.
I think I probably ended up just deleting the deployment and redeploying.
Do you mind sharing your pulumi code?
I actually remember there being 1 time I had this and the issue was caused by an integer overflow
b

bitter-dentist-28132

09/11/2019, 8:00 PM
c

cool-egg-852

09/11/2019, 8:01 PM
Have you tried without the transformations?
b

bitter-dentist-28132

09/11/2019, 8:02 PM
i added them because of this error, but sure i can try that.
c

cool-egg-852

09/11/2019, 8:03 PM
I’m not seeing anything that could possibly be wrong with it other than that, and I don’t see why that would cause it either, but worth a shot.
Other than that, I’d see if you can just remove the resource and recreate it if you need it fixed immediately. Otherwise file a pulumi issue. I’ve definitely run into this exact same issue previously.
Something causes pulumi to think it needs to be updated, but it doesn’t actually need an update, and thus it hangs.
And just to ask, what version of the chart are you using?
Given that you are not specifying a version, I’m not sure if pulumi updates the repo before executing or not.
Ah wait I see, 1.20.0
b

bitter-dentist-28132

09/11/2019, 8:19 PM
it's happened to me for both 1.17.x => 1.19.1 and from 1.19.1 => 1.20.0
last time i thought pulumi had just gotten into a bad state, so i blew away the cluster and recreated. now that it's happened again for the same reason, i know for sure it's a problem the update of the nginx-ingress chart.
also, it doesn't really matter if pulumi updates the repo, since i'm running in CI
the pulumi/pulumi image doesn't have helm, so i have to install it manually
so every time the CI runs, the repos should be up-to-date
c

cool-egg-852

09/11/2019, 8:25 PM
Ah. We’re not using the docker image for pulumi quite yet.
We’ll be moving to that as we replace Jenkins.
g

gorgeous-egg-16927

09/11/2019, 8:35 PM
Sorry about that, just responded on the issue. This turned out to be a bug with
extensions/v1beta1/Deployment
logic that only applied on updates.
c

cool-egg-852

09/11/2019, 8:44 PM
Fast as always.
😁 2
b

bitter-dentist-28132

09/11/2019, 8:53 PM
was that just released?
c

cool-egg-852

09/11/2019, 8:53 PM
Yep, about 14 minutes ago
b

bitter-dentist-28132

09/11/2019, 8:55 PM
damn, fast indeed. @gorgeous-egg-16927 think you can kick off a docker image build? 👼
c

cool-egg-852

09/11/2019, 8:56 PM
It was just a release of the provider, not sure a new docker image is needed is it?
b

bitter-dentist-28132

09/11/2019, 8:57 PM
oh, good point
g

gorgeous-egg-16927

09/11/2019, 8:57 PM
Right, you just need to update the provider version
5 Views