https://pulumi.com logo
Powered by
n

numerous-train-50906

09/08/2023, 8:57 PM
Does 
pulumi_eks
support creating a fully private EKS cluster (no public api access) with storage classes? Running into a strange issue:
Copy code
eks:index:VpcCni (staging-eks-vpc-cni):
    error: Command failed: kubectl apply -f /tmp/tmp-37930G3QLEJ0E0vMu.tmp
    Unable to connect to the server: net/http: TLS handshake timeout

  kubernetes:<http://storage.k8s.io/v1:StorageClass|storage.k8s.io/v1:StorageClass> (staging-eks-gp2):
    error: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://A5F09297FEECD3B9C32CA.gr7.ca-central-1.eks.amazonaws.com/openapi/v2?timeout=32s>": net/http: TLS handshake timeout

  kubernetes:core/v1:ConfigMap (staging-eks-nodeAccess):
    error: configured Kubernetes cluster is unreachable: unable to load schema information from the API server: Get "<https://A5F09297FEECD3B9C32CA.gr7.ca-central-1.eks.amazonaws.com/openapi/v2?timeout=32s>": net/http: TLS handshake timeout

Resources:
    21 unchanged

Duration: 7m42s
Here is the eks creation code:
Copy code
eks_cluster = eks.Cluster(
    resource_name=cluster_name,
    vpc_id=vpc_id,
    cluster_security_group=cluster_security_group,
    desired_capacity=desired_cluster_size,
    enabled_cluster_log_types=cluster_log_types,
    # Public subnets will be used for load balancers
    public_subnet_ids=public_subnet_ids,
    # Private subnets will be used for worker nodes
    private_subnet_ids=private_subnet_ids,
    # Do not give worker nodes a public IP address
    node_associate_public_ip_address=False,
    # Change configuration values to change any of the following settings
    name=cluster_name,
    skip_default_node_group=True,
    storage_classes={
        storage_class["name"]: storage_class["args"]
        for storage_class in storage_classes
    },
    tags={"Name": f"{name_tag}-eks"},
    version=eks_version,
    # Uncomment the next two lines for private cluster (VPN access required)
    endpoint_private_access=True,
    endpoint_public_access=False
)
b

billowy-army-68599

09/08/2023, 8:59 PM
yes, but you need to have the ability to contact the API for the kubernetes cluster from where your Pulumi program runs
ie a VPN
it’s really a network problem, rather than a pulumi program 🙂
n

numerous-train-50906

09/08/2023, 9:00 PM
confirmed, able to access from my laptop:
Copy code
$ telnet <http://a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com|a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com> 443
Trying 10.36.2.223...
Connected to <http://a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com|a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com>.
Escape character is '^]'.
b

billowy-army-68599

09/08/2023, 9:01 PM
can you 
kubectl
it?
n

numerous-train-50906

09/08/2023, 9:03 PM
unable to, same issue
Copy code
$ KUBECONFIG=~/.kube/eks-works.yaml kubectl get nodes
Unable to connect to the server: net/http: TLS handshake timeout
b

billowy-army-68599

09/08/2023, 9:03 PM
do you use a http proxy?
n

numerous-train-50906

09/08/2023, 9:04 PM
none, that i'm aware. I'm on a company vpn connection though
b

billowy-army-68599

09/08/2023, 9:04 PM
can you show me 
cluster_security_group
definition?
n

numerous-train-50906

09/08/2023, 9:05 PM
sure, from code or aws_console?
Copy code
def create_security_group(vpc_id: str, sg_name: str) -> aws.ec2.SecurityGroup:
    security_group = aws.ec2.SecurityGroup(
        f"{sg_name}-securityGroup",
        name_prefix=sg_name,
        description="EKS-Cluster-security-group",
        vpc_id=vpc_id,
    )
    security_group.id.apply(
                lambda sg_id: aws.ec2.SecurityGroupRule(
                    f"{sg_id}-https-443",
                    description="allow inbound 443 access from connected networks",
                    type="ingress",
                    from_port=443,
                    to_port=443,
                    protocol="tcp",
                    cidr_blocks=["0.0.0.0/0"],
                    security_group_id=sg_id,
                    opts=pulumi.ResourceOptions(depends_on=[security_group]),
                )
            )
    return security_group


cluster_security_group = create_security_group(vpc_id, f"{name_tag}-cluster-security-group")
b

billowy-army-68599

09/08/2023, 9:08 PM
hmm 🤔 well, the issue here is that the eks provider uses kubectl to make some changes, and ultimately there’s either a network error or a proxy intercepting. I imagine you’ll get the same response if you try to 
curl
the k8s api too. i’d double check your network connectivity (I know the telnet works, so something may be amiss elsewhere)
but it definitely is supported
n

numerous-train-50906

09/08/2023, 9:11 PM
yeah curl fails too
Copy code
$ curl -k -v <https://a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com>
*   Trying 10.36.2.223:443...
* Connected to <http://a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com|a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com> (10.36.2.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to <http://a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com:443|a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com:443> 
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to <http://a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com:443|a5f09297feecd3b9c32ca.gr7.ca-central-1.eks.amazonaws.com:443>
b

billowy-army-68599

09/08/2023, 9:13 PM
can you check your security group in the console and make sure that rule applied?
n

numerous-train-50906

09/08/2023, 9:15 PM
looks like they were:
b

billowy-army-68599

09/08/2023, 9:22 PM
hmm 🤔
n

numerous-train-50906

09/08/2023, 9:23 PM
i even opened up ephemeral ports (0-65535), based on a colleagues suggestion, but still the issue persists. I'm using a AWS vpn connection, via a transit gateway.
c

cuddly-computer-18851

09/09/2023, 12:11 AM
Check your NACLs as well, especially if TGW is in play.
n

numerous-train-50906

09/11/2023, 4:39 PM
@cuddly-computer-18851 - thanks, I did, as I'm able to connect locally via telnet.
Just to close the loop on this issue. It turned out that the network where 
pulumi up
was run didn't have full access to the VPN connection.
Thanks jaxxstorm, your hunch was correct.
Powered by