https://pulumi.com logo
#kubernetes
Title
# kubernetes
b

brief-church-51632

02/22/2023, 5:06 PM
I'm struggling with create and destroy order for my k8s stack. I'm trying to build a stack that launches eks, then installs the aws LB controller using helm (composite resource), and then create an ingress, which uses the controller to create an ALB. The problem is that I cannot get the ingress creation to wait until the Deployment deployed by the Helm chart is created and initialized, and so the ingress creation often fails. And I cannot get the Deployment teardown to wait until the Ingress is torn down, so the Ingress delete usually blocks indefinitely. I have code that looks like this:
Copy code
lb_controller_chart = k8s.helm.v3.Chart(
    "albctl",
    k8s.helm.v3.ChartOpts(
        chart="aws-load-balancer-controller",
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="<https://aws.github.io/eks-charts>"
        ),
        namespace=alb_controller_sa_namespace,
        values={
            "region": "us-west-2",
            "serviceAccount": {
                "name": alb_controller_sa,
                "create": False,
            },
            "vpcId": eks_cluster.eks_cluster.vpc_config.vpc_id.apply(lambda c: c),
            "clusterName": eks_cluster.eks_cluster.name.apply(lambda c: c),
        },
    ),
    opts=pulumi.ResourceOptions(provider=eks_provider)
)
...
ingress = k8s.networking.v1.Ingress(
    "myingress",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="myingress",
        annotations={
            "<http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>": "ip",
            "<http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>": "[{\"HTTP\": 80}]",
            "<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>": "internet-facing",
            "<http://alb.ingress.kubernetes.io/subnets|alb.ingress.kubernetes.io/subnets>": eks_vpc.public_subnet_ids.apply(lambda ids: ", ".join(ids)),
            "<http://alb.ingress.kubernetes.io/backend-protocol|alb.ingress.kubernetes.io/backend-protocol>": "HTTP"
        }
    ),
    spec=k8s.networking.v1.IngressSpecArgs(
        ingress_class_name="alb",
        rules=[k8s.networking.v1.IngressRuleArgs(
            host="*.helloworld.responsive.dev",
            http=k8s.networking.v1.HTTPIngressRuleValueArgs(
                paths=[k8s.networking.v1.HTTPIngressPathArgs(
                    backend=k8s.networking.v1.IngressBackendArgs(
                        service=k8s.networking.v1.IngressServiceBackendArgs(
                            name="helloworld",
                            port=k8s.networking.v1.ServiceBackendPortArgs(number=8081)
                        )
                    ),
                    path="/",
                    path_type="Prefix"
                )]
            )
        )]
    ),
    opts=pulumi.ResourceOptions(
        provider=eks_provider, depends_on=[lb_controller_chart])
)
At create time, I need the ingress to only get created after the deployment resource that is part of the helm chart is created. However, often times it does not wait and the ingress creation fails. Similarly, at delete time the deployment in the helm chart is deleted before the ingress, and the ingress deletion hangs. I thought setting
depends_on
would cover me here, but it doesn't seem to be working. My pulumi version is
v3.55.0
pulumi_kubernetes is at
pulumi_kubernetes-3.24.1
s

steep-toddler-94095

02/22/2023, 5:37 PM
b

brief-church-51632

02/22/2023, 5:54 PM
trying it now. Interestingly I don't see that example in the python docs
doesn't work. at first
pulumi up
seems to work, but it isn't deploying everything in the helm chart. In particular it misses a secret that the controller needs. Then, when I try to run
pulumi up
again I get:
Copy code
pulumi:pulumi:Stack (sindri-devel):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/opt/homebrew/bin/pulumi-language-python-exec", line 192, in <module>
        loop.run_until_complete(coro)
      File "/opt/homebrew/Cellar/python@3.11/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
        return future.result()
               ^^^^^^^^^^^^^^^
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/stack.py", line 136, in run_in_stack
        await run_pulumi_func(lambda: Stack(func))
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/stack.py", line 51, in run_pulumi_func
        await wait_for_rpcs()
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/stack.py", line 120, in wait_for_rpcs
        raise exception
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/rpc_manager.py", line 71, in rpc_wrapper
        result = await rpc
                 ^^^^^^^^^
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/resource.py", line 811, in do_register
        resolver = await prepare_resource(res, ty, custom, remote, props, opts, typ)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/resource.py", line 161, in prepare_resource
        explicit_urn_dependencies = await _resolve_depends_on_urns(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/Users/rohan/responsive/sindri/venv/lib/python3.11/site-packages/pulumi/runtime/resource.py", line 1171, in _resolve_depends_on_urns
        all_deps.add(direct_dep)
    TypeError: unhashable type: 'list'
s

steep-toddler-94095

02/22/2023, 7:24 PM
hmm looks like a bug. You can submit a GitHub Issue but in the meantime does this work for you? It was the workaround in the past:
Copy code
depends_on=b_controller_chart.resources.apply(lambda resources: list(resources.values())),
https://github.com/pulumi/pulumi-kubernetes/issues/861#issuecomment-901862700
b

brief-church-51632

02/22/2023, 7:26 PM
ooh I'll try that next. Right now I'm trying:
Copy code
depends_on=[lb_controller_chart.get_resource(
            "apps/v1/Deployment", "kube-system/albctl-aws-load-balancer-controller")]
thanks
your suggestion has worked so far (done a couple of up/destroy cycles). thanks!
g

gorgeous-minister-41131

02/24/2023, 12:29 AM
Nice one - TIL^ - I haven't had to do a Chart depending on a Chart but I imagine the workaround is because Pulumi isn't natively iterating thru all the resources, just ensuring the original Chart resource is created/ready in the Stack State and doesn't check all of the children?
s

steep-toddler-94095

02/24/2023, 12:54 AM
yup pretty much
4 Views