dazzling-oxygen-84405
02/15/2023, 1:52 PMwarning: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://172.18.3.244:6443/openapi/v2?timeout=32s>": dial tcp 172.18.3.244:6443: i/o timeout
error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster has been deleted, you can edit the pulumi state to remove this resource
This is mostly expected, because the preview is running in CI which doesn't have access to the VPN that the cluster is running on. However, this setup worked for several months, until sometime between two weeks ago and today, so I'm curious if anything has changed here?
Downgrading the Pulumi version doesn't seem to make a difference, and we haven't changed the version of the kubernetes provider.billowy-army-68599
02/15/2023, 2:07 PMdazzling-oxygen-84405
02/15/2023, 2:12 PMsteep-toddler-94095
02/15/2023, 5:06 PMnc -v 172.18.3.244 6443
172 is a private ip space so your runners either need to exist in the same VPC or something like a VPN. If you don't have that type of setup, it won't work and wouldn't have ever worked.
I would double check if things changed about your network, agents, or cluster recently. Even things like if the cluster used to have a publicly accessible API serverdazzling-oxygen-84405
02/15/2023, 7:17 PMIf you don't have that type of setup, it won't work and wouldn't have ever worked..Exactly, I'm 99% certain that github actions never had access to the Kubernetes API. The fact remains that this did work, for several months, until a week or two ago. It's a private repo, so I can't share a link to the action run/PR, but it absolutely produced previews of changes to this cluster. The only possible explanation is that
pulumi preview
was previously not contacting the API server at all, and just diffing the saved state with what running the stack gives (which, unexpectedly, seems to have been possible without contacting the API server). And recently, something changed which caused a preview to require the API server.
The long-term solution is to set up a VPN for our runners, and I'm working on this, I was just curious if anyone was aware of something that changed on the Pulumi side that now requires access to the API server.gorgeous-minister-41131
02/15/2023, 7:21 PM.kubeconfig
somewhere?kubectl config current-context
, is there a context defined? if so, drill into that and see if it's defining a server that matches that host?steep-toddler-94095
02/15/2023, 7:23 PMdazzling-oxygen-84405
02/15/2023, 7:25 PMkubeconfig
option.
I'm not setting the --refresh
flag in CI. I tried setting it to false
locally (looks like true
is the default) and running it while disconnected from the VPN, but this shows the same error.gorgeous-minister-41131
02/15/2023, 7:27 PMpulumi update
with live cluster access, and the CI is trying to use the same state?dazzling-oxygen-84405
02/15/2023, 7:30 PMconsidering serverSideAPply is going to become the provider default in the future, not having access to a k8s cluster, even a mock one, is going to make the provider preview relatively useless going forwardFair enough. I'm really looking forward to the patch functionality - that'll make some parts of our stack a little tidier. I'll get working on the VPN setup. That's definitely the way forward, I was just hoping for a flag I could toggle to work around this for a day or two. But no worries if not.
are you using an explicit provider with ResourceOptions?Yes, I'm only setting
kubeconfig
to the provider.gorgeous-minister-41131
02/15/2023, 7:31 PMdazzling-oxygen-84405
02/15/2023, 7:33 PMsomeone ran aYes, our normal workflow is: 1. Runwith live cluster access, and the CI is trying to use the same state?pulumi update
pulumi preview
from CI (this is the thing that's no longer working)
2. If the diff looks good, merge the PR
3. Run pulumi update
from a machine that has access to the VPN
This has been working for some time, nothing about this has changed recently.gorgeous-minister-41131
02/15/2023, 7:33 PM172.18.3.244
dazzling-oxygen-84405
02/15/2023, 7:34 PMsteep-toddler-94095
02/15/2023, 7:42 PMgorgeous-minister-41131
02/15/2023, 7:43 PMkubeconfig
param you're passing in looks like, or if it is too sensitive to shareRelease()
resource still tries to reach the cluster, live.. and all Release does, AFAIK, is just leverage the built-in Helm v3 bindings.. perhaps these have changed their behavior as well.dazzling-oxygen-84405
02/15/2023, 7:56 PMRelease
resources, but they haven't changed recently. The kubeconfig
is below (secrets snipped) - afaik nothing unusual about it.
{
"apiVersion": "v1",
"kind": "Config",
"clusters": [
{
"name": "on-prem",
"cluster": {
"certificate-authority-data": "...secret...",
"server": "<https://172.18.3.244:6443>"
}
}
],
"users": [
{
"name": "pulumi",
"user": {
"client-certificate-data": "...secret...",
"client-key-data": "...secret..."
}
}
],
"contexts": [
{
"name": "pulumi@on-prem",
"context": {
"cluster": "on-prem",
"user": "pulumi"
}
}
],
"current-context": "pulumi@on-prem"
}
actions-runner-controller
yet, but I'll add this to the list of advantages it has 🙂gorgeous-minister-41131
02/15/2023, 7:57 PMsteep-toddler-94095
02/15/2023, 8:03 PMgorgeous-minister-41131
02/15/2023, 8:04 PMPreviewing update (prd-bravo):
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:prd-bravo::k8s-aws-auth-config::pulumi:pulumi:Stack::k8s-aws-auth-config-prd-bravo]
> kubernetes:core/v1:Namespace: (read)
[id=kube-system]
[urn=urn:pulumi:prd-bravo::k8s-aws-auth-config::kubernetes:core/v1:Namespace::kube-system]
[provider=urn:pulumi:prd-bravo::k8s-aws-auth-config::pulumi:providers:kubernetes::eks_prd-bravo::8a15f8ed-f5ed-4537-a7bf-41d1aed334b8]
warning: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "https://<REDACTED>.sk1.us-gov-west-1.eks.amazonaws.com1/openapi/v2?timeout=32s": dial tcp: lookup <REDACTED>.sk1.us-gov-west-1.eks.amazonaws.com1 on 10.0.0.2:53: no such host
error: Preview failed: failed to read resource state due to unreachable cluster. If the cluster has been deleted, you can edit the pulumi state to remove this resource or retry with the PULUMI_K8S_DELETE_UNREACHABLE environment variable set to true.
error: preview failed
Resources:
2 unchanged
warning: A new version of Pulumi is available. To upgrade from version '3.53.0' to '3.55.0', visit <https://p>
warning: configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file. Make sure you have:
• set up the provider as per <https://www.pulumi.com/registry/packages/kubernetes/installation-configuration/>
server
in the context is definitely injecting private IPs. You mentioned this value was coming from an output of another thing, perhaps another Pulumi state/project that is exporting the API server endpoint? As Mike indicated, did someone make this endpoint private [which is a good idea of course], and now the output contains this IP instead of the publicly accessible hostname that 'worked' before? Do you use EKS?dazzling-oxygen-84405
02/15/2023, 8:15 PMgorgeous-minister-41131
02/15/2023, 8:18 PMclusterUnreachable
hasn't changed in 2-4 years, and the only recent change made to that provider for that exception is the mention of the env var I put above. 🤷// We use this information to read the live version of a Kubernetes resource. This is sometimes
// then checkpointed (e.g., in the case of `refresh`). Specifically:
//
// * The return is formatted as a "checkpoint object", i.e., an object of the form
// {inputs: {...}, live: {...}}. This is important both for `Diff` and for `Update`. See
// comments in those methods for details.
//