sparse-intern-71089
10/06/2023, 6:05 AMeager-coat-38141
10/06/2023, 6:08 AMprovider "digitalocean" {}
resource "null_resource" "previous" {}
// Digitalocean load balancer is not managed by terraform.
// We wait for the loadbalancer to be ready so we can read its ip and output it.
resource "time_sleep" "wait_for_load_balancer" {
depends_on = [null_resource.previous]
create_duration = "180s"
}
data "digitalocean_loadbalancer" "loadbalancer" {
depends_on = [time_sleep.wait_for_load_balancer]
id = var.id
}
With this I get a data object with the load balancer which can output its IP and other stuff.
Of course the entire script pauses for 3 minutes until the IP is available.eager-coat-38141
10/06/2023, 6:16 AMdry-keyboard-94795
10/06/2023, 6:21 AM.get()
, but if you access the ip address via the status of the service, it should wait for you.
Example of what the status object looks like: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancerdry-keyboard-94795
10/06/2023, 6:24 AMlb_ip = existing_service.status.loadBalancer.ingress[0].ip
dry-keyboard-94795
10/06/2023, 6:45 AMdry-keyboard-94795
10/06/2023, 6:46 AMsleep
is missing from the pulumi time provider wrapper, so you can't replicate your TF directly: https://www.pulumi.com/registry/packages/time/
This should also have an issue raisedeager-coat-38141
10/06/2023, 6:47 AM"<http://service.beta.kubernetes.io/do-loadbalancer-hostname|service.beta.kubernetes.io/do-loadbalancer-hostname>"
with the final DNS hostname there.
So the status of the service will never show me the ip, it will show me the host name i’ve set up beforeeager-coat-38141
10/06/2023, 6:49 AMInterestingly,Yep, sleep is missing altogether, but I seem to understand it’s because we should not be relying on sleep but on other more asynchronous friendly approaches. Problem is I am missing something here and I cannot really understand what 😅 I am also pretty sure I am not the first one with this problem…is missing from the pulumi time provider wrapper, so you can’t replicate your TF directlysleep
dry-keyboard-94795
10/06/2023, 6:50 AMlocal.Command
to run sleep yourself, and have the DO LB resource depend on itdry-keyboard-94795
10/06/2023, 6:52 AMdry-keyboard-94795
10/06/2023, 6:54 AMShould not rely on sleep
The real world is rarely so kind as to allow the ideal 🤣
dry-keyboard-94795
10/06/2023, 6:59 AM.get()
, you can pass in opts=pulumi.ResourceOptions(depends_on=[sleeper])
Where sleeper is this resource: https://www.pulumi.com/registry/packages/command/api-docs/local/command/eager-coat-38141
10/06/2023, 7:05 AMdry-keyboard-94795
10/06/2023, 7:07 AMeager-coat-38141
10/06/2023, 7:09 AM# Prepare the kubernetes provider
kubeconfig = cluster.kube_configs[0].raw_config
kubernetes_provider = k8s_provider("kubernetes_provider", kubeconfig=kubeconfig)
opts = pulumi.ResourceOptions(
depends_on=[cluster, kubernetes_provider],
provider=kubernetes_provider,
)
def set_load_balancer_annotation(obj, opts):
"""If the kubernetes object is of type LoadBalancer, adds an annotation to it"""
if obj["kind"] == "Service" and obj["apiVersion"] == "v1":
try:
t = obj["spec"]["type"]
if t == "LoadBalancer":
obj["metadata"]["annotations"][
"<http://service.beta.kubernetes.io/do-loadbalancer-hostname|service.beta.kubernetes.io/do-loadbalancer-hostname>"
] = DUMMY_SERVICE_URL
except KeyError:
pass
# ingress nginx controller
ingress_nginx_workload = ConfigFile(
"ingress-nginx",
opts=opts,
skip_await=True,
transformations=[set_load_balancer_annotation],
file="kubernetes_manifests/ingress-nginx/v1.5.1/deploy.yaml",
)
dry-keyboard-94795
10/06/2023, 7:11 AMeager-coat-38141
10/06/2023, 7:13 AMdry-keyboard-94795
10/06/2023, 7:14 AMdry-keyboard-94795
10/06/2023, 7:21 AMConfigFile().get_resource()
, which let's you access the service directly.
So existing_service = ingress_nginx_workload.get_resource("core/v1/Service", "ingress-nginx-controller", namespace="ingress-nginx")
eager-coat-38141
10/06/2023, 7:34 AMskip_await
. at this stage, when installing the nginx controller, the deployment times out because it waits for application pods to actually send traffic to.
The application workloads are however installed on the cluster at a later step and outside of Pulumi, with ArgoCD.dry-keyboard-94795
10/06/2023, 7:39 AMskip_await
dry-keyboard-94795
10/06/2023, 7:48 AMdry-keyboard-94795
10/06/2023, 8:49 AMeager-coat-38141
10/06/2023, 9:44 AM# Get an existing Kubernetes Service
opts_existing_service = pulumi.ResourceOptions(
depends_on=[cluster, kubernetes_provider, ingress_nginx_workload],
provider=kubernetes_provider,
)
existing_service = Service.get(
"existing-service",
f"ingress-nginx/ingress-nginx-controller",
opts=opts_existing_service,
)
# Read the load balancer ID
load_balancer_id = existing_service.metadata["annotations"]["<http://kubernetes.digitalocean.com/load-balancer-id|kubernetes.digitalocean.com/load-balancer-id>"]
wait_cmd = command.local.Command("wait_cmd",
create="sleep 180",
delete="true",
opts=pulumi.ResourceOptions(depends_on=[existing_service])
)
# Grab the DO load balancer
load_balancer = digitalocean.LoadBalancer.get(
"ingress-nginx-load-balancer",
id=load_balancer_id,
opts=pulumi.ResourceOptions(depends_on=[wait_cmd])
)
# Grab the IP from the DO load balancer
load_balancer_ip = pulumi.Output.from_input(load_balancer.ip).apply(lambda ip: ip if ip else 'IP not assigned yet')
pulumi.export("load_balancer_id", load_balancer_id)
pulumi.export("load_balancer_ip", load_balancer_ip)
I’d prefer using a pulumi provider for sleep like Terraform has, but this also works.
Thanks again, @dry-keyboard-94795 🤝dry-keyboard-94795
10/06/2023, 9:48 AMrhythmic-secretary-1287
10/06/2023, 11:37 AMdry-keyboard-94795
10/06/2023, 11:42 AMrhythmic-secretary-1287
10/06/2023, 11:44 AMeager-coat-38141
10/06/2023, 12:04 PMtime_sleep
, at least that’s how I dealt with TGW in the past.
Im surprised a provider for waiting doesn’t exist out of the box for Pulumi.rhythmic-secretary-1287
10/06/2023, 12:09 PMbillowy-army-68599
eager-coat-38141
10/06/2023, 1:09 PMdry-keyboard-94795
10/06/2023, 1:19 PMbillowy-army-68599
dry-keyboard-94795
10/06/2023, 2:23 PMbillowy-army-68599
dry-keyboard-94795
10/06/2023, 2:42 PMConfigFile
) doing a selective skipAwait on the Deployment so that the Service will await its endpoints + status is the way to go, anyway.
Unsure if it'll work with the pods failing healthchecks though.
@eager-coat-38141, up for trying some code if I refactor yours?eager-coat-38141
10/07/2023, 2:19 PMingress-nginx/ingress-nginx-controller
and ingress-nginx/ingress-nginx-controller-admission
are the ones waiting for pods.
+ └─ kubernetes:core/v1:Service ingress-nginx/ingress-nginx-controller creating (416s)... [1/3] Finding Pods to direct traffic to
...
It feels like adding the skip await annotation to them will still cause the original issue with the Load balancer IP though.
In any case I tried that and so far bumped into another issue where one of the ingress-nginx jobs is unable to complete.
Kind of feel like the sleep command resource solution yielded the most results so far 😅
I will also try out @billowy-army-68599’s solution and see where I end updry-keyboard-94795
10/10/2023, 12:36 PMv0.0.16