https://pulumi.com logo
Join Slack
Powered by
Hi all, I am trying to deploy a pulumi program fr...
# general
h
Hi all, I am trying to deploy a pulumi program from a private github repo with pulumi kubernetes operator (version 2.0.0). I have create ssh deploy key but when I am trying to install the program, my stack is always going stalled because of:
Copy code
"message":"listing: unable to find any valid known_hosts file, set SSH_KNOWN_HOSTS env variable
However, I have provided the SSH_KNOWN_HOSTS to both to the helm release and in the stack manifest to deploy my program? Anyone as ever facing such issue? More details in the 🧵 .
I have provide a SSH_KNOWN_HOSTS to the operator controller when deploying it with helm (the operator is deployed with terraform 😓 ):
Copy code
resource "helm_release" "pulumi_kubernetes_operator" {
  name       = "pulumi-kubernetes-operator"
  namespace  = var.namespace
  chart      = "pulumi-kubernetes-operator"
  repository = "<oci://ghcr.io/pulumi/helm-charts>"
  version    = "2.0.0"


  values = [
    yamlencode({
      serviceAccount = {
        create = false
        name   = "some-service-acount"
      }
      extraEnv = [
        {
          name = "SSH_KNOWN_HOSTS"
          value = <<-EOT
            <http://github.com|github.com> ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB....
            <http://github.com|github.com> ecdsa-sha2-nistp256 AAAAE2....
            <http://github.com|github.com> ssh-ed25519 AAAAC3NzaC1lZ....
          EOT
        },
        {
          name = "SSH_PRIVATE_KEY"
          valueFrom = {
            secretKeyRef = {
              name = "pko-github-ssh-key"
              key  = "SSH_PRIVATE_KEY"
            }
          }
        }
      ],
      controller = {
        logLevel = "debug"
      }
      resources = {
        limits = {
          memory = "1Gi"
        }
        requests = {
          memory = "512Mi"
        }
      }
    })
  ]
}"
....
And in the stack in the envRefs and workspaceTemplate:
Copy code
apiVersion: <http://pulumi.com/v1|pulumi.com/v1>
kind: Stack
metadata:
  name: pulumi-poc-stack
spec:
  stack: "organization/test-program/dev"
  gitAuth:
    sshAuth:
      sshPrivateKey:
        type: Secret
        secret:
          name: pko-github-ssh-key
          key: SSH_PRIVATE_KEY

  projectRepo: "git@github.com:...."
  branch: "REVEAL-9081-poc"
  envRefs:
    SSH_KNOWN_HOSTS:
      type: Secret
      secret:
        name: ssh-known-hosts-pulumi-operator
        key: SSH_KNOWN_HOSTS
    GITHUB_TOKEN:
      type: Secret
      secret:
        name: pko-github-ssh-key
        key: GITHUB_TOKEN
  destroyOnFinalize: true
  backend: "s3://..."
  refresh: true
  resyncFrequencySeconds: 60
  serviceAccountName: <service account>
  shallow: true
  workspaceTemplate:
    spec:
      env:
        - name: SSH_KNOWN_HOSTS
          valueFrom:
            secretKeyRef:
              name: ssh-known-hosts-pulumi-operator
              key: SSH_KNOWN_HOSTS
        - name: SSH_PRIVATE_KEY
          valueFrom:
            secretKeyRef:
              name: pko-github-ssh-key
              key: SSH_PRIVATE_KEY

      pulumiLogLevel: 10
      resources:
        requests:
          memory: 1Gi
        limits:
          memory: 2Gi
      podTemplate:
        spec:
          containers:
            - name: pulumi
              imagePullPolicy: Always
              resources:
                requests:
                  memory: 1Gi
                limits:
                  memory: 2Gi
          initContainers:
            - name: fetch
              resources:
                requests:
                  memory: 1Gi
                limits:
                  memory: 2Gi
The workspace pod is not even kicking in so my guess is that the issue is at the controller . From the log I have:
Copy code
INFO	Status updated	{"controller": "stack-controller", "namespace": "test", "name": "pulumi-poc-stack", "reconcileID": "9ef6a0d7-1f1f-4bfb-aafe-3fb381f154ef", "revision": "2663948564", "observedGeneration": 0, "observedReconcileRequest": "", "lastUpdate": null, "currentUpdate": null, "conditions": [{"type":"Ready","status":"False","lastTransitionTime":"2025-08-06T14:24:22Z","reason":"NotReadyStalled","message":"reconciliation is stalled"},{"type":"Stalled","status":"True","lastTransitionTime":"2025-08-06T14:24:22Z","reason":"SourceUnavailable","message":"listing: unable to find any valid known_hosts file, set SSH_KNOWN_HOSTS env variable"}]}
h
That does seem odd. Perhaps something wonky in 2.0.0? There are quite a few fixes in 2.1.0. Something to possibly try?
👀 1
For further comparison, I specify SSH_KNOWN_HOSTS in the operator deployment (also Helm installed), but in the stack(s) I'm only specifying it in the 
fetch
container (as opposed to the workspaceTemplate as you have it). Unsure why it all fleshed out that way, but it IS working... so 🤷
h
Ok, I give give a try with 2.1.0 and the 
fetch
container. For the 
fetch
, the workspace pod is not even reaching the point of being launched but in any case, I'll try it.
h
Oh. I think I see your problem. SSH_KNOWN_HOSTS wants a filename, not the contents.
h
Ak ok, so I need to create a file with the known hosts and set the name of the file in SSH_KNOWN_HOSTS
@hallowed-shoe-53735 Thanks, it works 🙏🎉 ! For future reference: In the helm value, I added:
Copy code
extraEnv = [
        {
          name = "SSH_KNOWN_HOSTS"
          value = "/etc/ssh-known-hosts/known_hosts"
        },
        {
          name = "SSH_PRIVATE_KEY"
          valueFrom = {
            secretKeyRef = {
              name = "pko-github-ssh-key"
              key  = "SSH_PRIVATE_KEY"
            }
          }
        }
      ],
      extraVolumeMounts = [
        {
          name = "ssh-known-hosts-volume",
          mountPath = "/etc/ssh-known-hosts",
          readOnly = true,
        }
      ],
      extraVolumes = [
        {
          name = "ssh-known-hosts-volume",
          secret = {
            secretName = "ssh-known-hosts-pulumi-operator"
          }
        }
      ],
And in the stack resource:
Copy code
podTemplate:
        spec:
          containers:
            - name: pulumi
              imagePullPolicy: Always
              resources:
                requests:
                  memory: 1Gi
                limits:
                  memory: 2Gi
          initContainers:
            - name: fetch
              env:
                - name: SSH_KNOWN_HOSTS
                  value: /etc/ssh-known-hosts/known_hosts
              volumeMounts:
                - name: ssh-known-hosts-volume
                  mountPath: /etc/ssh-known-hosts
                  readOnly: true
              resources:
                requests:
                  memory: 1Gi
                limits:
                  memory: 2Gi
          volumes:
            - name: ssh-known-hosts-volume
              secret:
                secretName: ssh-known-hosts-pulumi-operator
Where ssh-known-hosts-pulumi-operator contains :
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: ssh-known-hosts-pulumi-operator
  namespace: ml
type: Opaque
stringData:
  known_hosts: |
    <http://github.com|github.com> ssh-rsa AAAAB3N...
    <http://github.com|github.com> ecdsa-sha2-nistp256 AAAAE2...
    <http://github.com|github.com> ssh-ed25519 AAAAC3Nza...
☝️ Which you can get from 
ssh-keyscan <http://github.com|github.com>
h
🎉 Good deed for the day ✅ 😜
5 Views
Open in Slack
PreviousNext
Powered by