eager-coat-38141
10/06/2023, 6:05 AMLoadBalancer
, after which the cloud provider automatically provisions a digitalocean load balancer for the cluster.
Unfortunately this is is outside of Pulumi’s control, so I wrote a snippet which fetches the load balancer id from the service’s annotation, queries the digitalocean api for the load balancer and outputs its IP.
The issue is that once the service of type LoadBalancer
is created and the load balancer is deployed, it takes up to 3~ minutes for it to be assigned an IP.
I need that IP for subsequent steps in the same script, so I have been trying to find ways to get Pulumi to wait for the IP to be available.
Here is the code i’ve got so far (suggested by several iterations with Pulumi AI):
# Get an existing Kubernetes Service
opts_existing_service = pulumi.ResourceOptions(
depends_on=[cluster, kubernetes_provider, ingress_nginx_workload],
provider=kubernetes_provider,
)
existing_service = Service.get(
"existing-service",
f"ingress-nginx/ingress-nginx-controller",
opts=opts_existing_service,
)
# Read the load balancer ID
load_balancer_id = existing_service.metadata["annotations"]["<http://kubernetes.digitalocean.com/load-balancer-id|kubernetes.digitalocean.com/load-balancer-id>"]
# Grab the DO load balancer
load_balancer = digitalocean.LoadBalancer.get("ingress-nginx-load-balancer", id=load_balancer_id)
# Grab the IP from the DO load balancer
load_balancer_ip = pulumi.Output.from_input(load_balancer.ip).apply(lambda ip: ip if ip else 'IP not assigned yet')
pulumi.export("load_balancer_id", load_balancer_id)
pulumi.export("load_balancer_ip", load_balancer_ip)
What I get from this is always an output saying load_balancer_ip: "IP not assigned yet"
, because the Pulumi script finishes running before the digitalocean load balancer has an ip address.
I understand that using time.sleep
or while
loops to continuously poll for the IP is not best practice, plus that didn’t really work for me due to the asynchronous nature of the program.
What is the correct way for me to wait for the load balancer IP to be available and continue the program, while using Pulumi tools?provider "digitalocean" {}
resource "null_resource" "previous" {}
// Digitalocean load balancer is not managed by terraform.
// We wait for the loadbalancer to be ready so we can read its ip and output it.
resource "time_sleep" "wait_for_load_balancer" {
depends_on = [null_resource.previous]
create_duration = "180s"
}
data "digitalocean_loadbalancer" "loadbalancer" {
depends_on = [time_sleep.wait_for_load_balancer]
id = var.id
}
With this I get a data object with the load balancer which can output its IP and other stuff.
Of course the entire script pauses for 3 minutes until the IP is available.dry-keyboard-94795
10/06/2023, 6:21 AM.get()
, but if you access the ip address via the status of the service, it should wait for you.
Example of what the status object looks like: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancerlb_ip = existing_service.status.loadBalancer.ingress[0].ip
sleep
is missing from the pulumi time provider wrapper, so you can't replicate your TF directly: https://www.pulumi.com/registry/packages/time/
This should also have an issue raisedeager-coat-38141
10/06/2023, 6:47 AM"<http://service.beta.kubernetes.io/do-loadbalancer-hostname|service.beta.kubernetes.io/do-loadbalancer-hostname>"
with the final DNS hostname there.
So the status of the service will never show me the ip, it will show me the host name i’ve set up beforeInterestingly,Yep, sleep is missing altogether, but I seem to understand it’s because we should not be relying on sleep but on other more asynchronous friendly approaches. Problem is I am missing something here and I cannot really understand what 😅 I am also pretty sure I am not the first one with this problem…is missing from the pulumi time provider wrapper, so you can’t replicate your TF directlysleep
dry-keyboard-94795
10/06/2023, 6:50 AMlocal.Command
to run sleep yourself, and have the DO LB resource depend on itShould not rely on sleep
The real world is rarely so kind as to allow the ideal 🤣
.get()
, you can pass in opts=pulumi.ResourceOptions(depends_on=[sleeper])
Where sleeper is this resource: https://www.pulumi.com/registry/packages/command/api-docs/local/command/eager-coat-38141
10/06/2023, 7:05 AMdry-keyboard-94795
10/06/2023, 7:07 AMeager-coat-38141
10/06/2023, 7:09 AM# Prepare the kubernetes provider
kubeconfig = cluster.kube_configs[0].raw_config
kubernetes_provider = k8s_provider("kubernetes_provider", kubeconfig=kubeconfig)
opts = pulumi.ResourceOptions(
depends_on=[cluster, kubernetes_provider],
provider=kubernetes_provider,
)
def set_load_balancer_annotation(obj, opts):
"""If the kubernetes object is of type LoadBalancer, adds an annotation to it"""
if obj["kind"] == "Service" and obj["apiVersion"] == "v1":
try:
t = obj["spec"]["type"]
if t == "LoadBalancer":
obj["metadata"]["annotations"][
"<http://service.beta.kubernetes.io/do-loadbalancer-hostname|service.beta.kubernetes.io/do-loadbalancer-hostname>"
] = DUMMY_SERVICE_URL
except KeyError:
pass
# ingress nginx controller
ingress_nginx_workload = ConfigFile(
"ingress-nginx",
opts=opts,
skip_await=True,
transformations=[set_load_balancer_annotation],
file="kubernetes_manifests/ingress-nginx/v1.5.1/deploy.yaml",
)
dry-keyboard-94795
10/06/2023, 7:11 AMeager-coat-38141
10/06/2023, 7:13 AMdry-keyboard-94795
10/06/2023, 7:14 AMConfigFile().get_resource()
, which let's you access the service directly.
So existing_service = ingress_nginx_workload.get_resource("core/v1/Service", "ingress-nginx-controller", namespace="ingress-nginx")
eager-coat-38141
10/06/2023, 7:34 AMskip_await
. at this stage, when installing the nginx controller, the deployment times out because it waits for application pods to actually send traffic to.
The application workloads are however installed on the cluster at a later step and outside of Pulumi, with ArgoCD.dry-keyboard-94795
10/06/2023, 7:39 AMskip_await
eager-coat-38141
10/06/2023, 9:44 AM# Get an existing Kubernetes Service
opts_existing_service = pulumi.ResourceOptions(
depends_on=[cluster, kubernetes_provider, ingress_nginx_workload],
provider=kubernetes_provider,
)
existing_service = Service.get(
"existing-service",
f"ingress-nginx/ingress-nginx-controller",
opts=opts_existing_service,
)
# Read the load balancer ID
load_balancer_id = existing_service.metadata["annotations"]["<http://kubernetes.digitalocean.com/load-balancer-id|kubernetes.digitalocean.com/load-balancer-id>"]
wait_cmd = command.local.Command("wait_cmd",
create="sleep 180",
delete="true",
opts=pulumi.ResourceOptions(depends_on=[existing_service])
)
# Grab the DO load balancer
load_balancer = digitalocean.LoadBalancer.get(
"ingress-nginx-load-balancer",
id=load_balancer_id,
opts=pulumi.ResourceOptions(depends_on=[wait_cmd])
)
# Grab the IP from the DO load balancer
load_balancer_ip = pulumi.Output.from_input(load_balancer.ip).apply(lambda ip: ip if ip else 'IP not assigned yet')
pulumi.export("load_balancer_id", load_balancer_id)
pulumi.export("load_balancer_ip", load_balancer_ip)
I’d prefer using a pulumi provider for sleep like Terraform has, but this also works.
Thanks again, @dry-keyboard-94795 🤝dry-keyboard-94795
10/06/2023, 9:48 AMrhythmic-secretary-1287
10/06/2023, 11:37 AMdry-keyboard-94795
10/06/2023, 11:42 AMrhythmic-secretary-1287
10/06/2023, 11:44 AMeager-coat-38141
10/06/2023, 12:04 PMtime_sleep
, at least that’s how I dealt with TGW in the past.
Im surprised a provider for waiting doesn’t exist out of the box for Pulumi.rhythmic-secretary-1287
10/06/2023, 12:09 PMbillowy-army-68599
eager-coat-38141
10/06/2023, 1:09 PMdry-keyboard-94795
10/06/2023, 1:19 PMbillowy-army-68599
dry-keyboard-94795
10/06/2023, 2:23 PMbillowy-army-68599
dry-keyboard-94795
10/06/2023, 2:42 PMConfigFile
) doing a selective skipAwait on the Deployment so that the Service will await its endpoints + status is the way to go, anyway.
Unsure if it'll work with the pods failing healthchecks though.
@eager-coat-38141, up for trying some code if I refactor yours?eager-coat-38141
10/07/2023, 2:19 PMingress-nginx/ingress-nginx-controller
and ingress-nginx/ingress-nginx-controller-admission
are the ones waiting for pods.
+ └─ kubernetes:core/v1:Service ingress-nginx/ingress-nginx-controller creating (416s)... [1/3] Finding Pods to direct traffic to
...
It feels like adding the skip await annotation to them will still cause the original issue with the Load balancer IP though.
In any case I tried that and so far bumped into another issue where one of the ingress-nginx jobs is unable to complete.
Kind of feel like the sleep command resource solution yielded the most results so far 😅
I will also try out @billowy-army-68599’s solution and see where I end updry-keyboard-94795
10/10/2023, 12:36 PMv0.0.16