https://pulumi.com logo
#python
Title
# python
c

clean-engineer-75963

12/16/2019, 10:13 PM
I'm still struggling with port-forwarding to a pod during Pulumi setup, but am making progress. Here's the code I have currently:
Copy code
PORT_FORWARD = None

def get_vault_port_forward_address(vault_deployment_metadata):
    """Wait for a Vault Pod to become leader, port-forward to it, and return its address."""

    namespace = vault_deployment_metadata["namespace"]
    vault_component = vault_deployment_metadata["labels"]["component"]

    def get_ready_vault_pod():
        kubernetes.config.load_kube_config()
        client = kubernetes.client.CoreV1Api()

        while True:
            pods = client.list_namespaced_pod(
                namespace,
                label_selector=f"component={vault_component}",
            )

            for pod in pods.items:
                for status in pod.status.conditions:
                    if status.type == "Ready" and status.status == "True":
                        return pod.metadata.name

            time.sleep(2)

    # TODO: This port-forward will run for the remainder of the Pulumi program.
    # Find a way to tear it down as soon as we're done with it. The pulumi
    # programming model makes this tough.
    global PORT_FORWARD
    PORT_FORWARD = pexpect.spawn(
        "kubectl",
        ["-n", namespace, "port-forward", get_ready_vault_pod(), "8200:8200"],
        ignore_sighup=True)
    PORT_FORWARD.expect_exact("Forwarding from 127.0.0.1:8200 -> 8200\r\n")
    PORT_FORWARD.expect_exact("Forwarding from [::1]:8200 -> 8200\r\n")

    return "<https://127.0.0.1:8200>"

# Vault provider to be used via port-forward.
vault_provider = pulumi_vault.Provider(
    "vault-provider",
    address=vault_deployment.metadata.apply(
        get_vault_port_forward_address),
    skip_tls_verify=True,
    token="my-token",
)
I'm using a function in an apply to start the port-forward during initialization of the vault provider. Unfortunately I'm closing over a global variable so the port-forward persists outside of the apply.
The problem this has currently is that the port-forward starts up beautifully during the preview phase of
pulumi up
, and plans some changes based on vault state -- all correct.
But then, when the preview phase finishes, the port-forward dies, and is not started again when we move to the apply phase, so pulumi is unable to actually make the changes it previewed.
I'm guessing preview and apply take place in different processes? You can see that I tried to add
ignore_sighup=True
to the pexpect call to see if I could force it to become a zombie process. If that had worked, I would have then used
atexit
to register a function to terminate it before closing.
But that didn't work; it still dies when the preview finishes.
Is there a way I can force the address attribute of the vault provider to be evaluated again at apply time?
I spent a little time exploring multiprocessing and subprocess with options that force the process into a different process group. I tried the latter and it doesn't work (the preview phase hangs until you manually kill the port-forward process).
I think I'm going to abandon this port-forward. It just won't work. Eventually we were going to get to a place where we wanted to run Pulumi in a container inside the cluster anyway (since some of our clusters are on the other side of the world). I had hoped for the POC that I wouldn't have to tackle that, but it looks like it's not going to work at all otherwise.