https://pulumi.com logo
Join Slack
Powered by
Hi All, I have few questions. we are setting up EK...
# general
l
Hi All, I have few questions. we are setting up EKS AWS cluster. 1. What you suggest for monitoring and alerts. 2. What you suggest for Dashboard or generating arch diagrams for cluster. 3. Routing ? example I have cluster with 5 micro-service. earlier each deployed using 
'type: 'LoadBalancer'
I got the url using ELB automatically. those urls are very long url. I want to connect using Route53. so, I want to have this kind of urls. https://api.mydomain.com/service1 , https://api.mydomain.com/service2 , after looking some example, I changed type. it is now 
type: NodePort
. I may need to create 
k8s.networking.v1beta1.Ingress
and use 
aws-alb-ingress-controller
and route traffic. I tried and didn't worked because example was not having Route53 specific code. I don't know if it do automatically internally. all micro-services are working fine. deployed successfully. only the routing part is now remaining. So, please help me. 1. All services need 
NodePort
or 
LoadBalancer
2. What is good url scheme ? https://api_service1.mydomain.com or https://api.mydomain.com/service1 ? Kindly help and suggest. + Also, people suggest me to add API Gateway. where API Gateway comes into this picture?
s
I usually set route53 manually, out of laziness or paranoia since the load balancer usually doesn't change. You're right to use ALB, that will allow you to have path based routing, so /service1 and /service2 route to each indiviual service. So you'll want 
type: LoadBalancer
and a route53 record for the ALB to api.yourdomain.com (probably using their Alias functionality instead of specifying the load balancer as a CNAME) API gateway can stand in front of your stack, or replace ALB if you're only running Lambdas or ECS, but likely isn't needed for EKS. I don't think that ALB supports routing to different subdomains, but I could be incorrect.
l
@swift-painter-31084 so you mean, I need to setup one ALB and all pods at 
NodePort
? How to do this in pulumi ? Any example ??
is Istio help in this case? checking Istio videos.
Looking for Pulumi example to copy paste 🙂
s
No istio/app mesh needed! It has its perks, but overkill to start with. https://www.pulumi.com/docs/guides/crosswalk/aws/eks/ the @pulumi/EKS library is the way to go because Pulumi has gone ahead and set sane defaults on everything for an EKS deployment. That page has lots of examples
l
@swift-painter-31084 I think, there is only one example - https://www.pulumi.com/blog/kubernetes-ingress-with-aws-alb-ingress-controller-and-pulumi-crosswalk/ but I am following it but not worked out.
Copy code
2020-01-11T22:19:24.564521145Z E0111 22:19:24.564316       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to retrieval of subnets failed to resolve 2 qualified subnets. Subnets must contain the <http://kubernetes.io/cluster/|kubernetes.io/cluster/>\u003ccluster name\u003e tag with a value of shared or owned and the <http://kubernetes.io/role/elb|kubernetes.io/role/elb> tag signifying it should be used for ALBs Additionally, there must be at least 2 subnets with unique availability zones as required by ALBs. Either tag subnets to meet this requirement or use the subnets annotation on the ingress resource to explicitly call out what subnets to use for ALB creation. The subnets that did resolve were []"  "controller"="alb-ingress-controller" "request"={"Namespace":"syngenta-platform","Name":"syngenta-platform-router"}
s
hmm something is missing by step 3 as there are missing subnets
l
Yup, stuck on this from whole day ! may be my pods are not on default namespace. is that a problem
@swift-painter-31084 here is my code
f
@swift-painter-31084 How do you setup route53 manually with ALB?
@swift-painter-31084 The reason I ask is because the ALB ingress resource expects you to install externa-dns for that and we rather not.
@late-printer-99022 If you wanna attach a domain to your ALB use the external-dns CRD. Using a service with loadbalancer type doesn’t use ALB, it uses the old classic load balancer
s
Inside your hosted zone on route53, you'd create a new A record with the name of the subdomain, and an Alias as a target. If the ALB exists you should be able to select it from the dropdown menu of targets. PS. Here's the right way to do it 🙂 https://www.pulumi.com/docs/reference/pkg/nodejs/pulumi/aws/route53/#alias-record Also interesting, I didn't know that the loadbalancer type forces classic load balancer, that doesn't have path based routing.
f
@swift-painter-31084 Right, so far we’ve done that. The issue is getting the ALB address to add it with pulumi, that’s not been easy.
s
@late-printer-99022 Chris is right, check out https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html
f
The issue is connecting that tutorial to pulumi - I wish Pulumi had a built in way of installing the ALB or there was a way of using Pulumi to create and attach the ALB to EKS - in fact I quite hate that ALB is its whole separate thing.
s
That is definitely a downside of EKS in general. I'm not familar with the EKS library but it may have some sugar for that. In general though it would be a matter of passing the lb.arn string into the route53 record, and setting the loadbalancer under dependsOn in the options parameter ie.
Copy code
const myRecord = new aws.route53.Record("myRecord", {...}, { dependsOn: [myALB] });
f
That was the issue, ALB isn’t created by pulumi but but by these steps https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/controller/setup/
One way was installing ALB, and then describing the ingress and try to get the address
Haven’t tried that yet
s
I ran into challenges with route53 record dependencies, and ended up (for now) doing multiple deploys in a sort of hacky way. As we've evolved, we've added deletion protection to the ALB and ended up keeping it in its own stack with our other networking infrastructure, so separate deploys are now the default. We then register the ALB as a provider in our app stacks and add app specific listener rules there.
l
Copy code
E0111 22:57:45.744049       1 :0] kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to unable to fetch subnets due to FilterLimitExceeded: The maximum length for a filter value is 255 characters\n\tstatus code: 400, request id: 003754e4-6d2d-4912-9f73-2322f58e2d6f"  "controller"="alb-ingress-controller" "request"={"Namespace":"syngenta-platform","Name":"syngenta-platform-router"}
f
That doesn’t tell me much dude.
You might wanna gist your configs somewhere.
l
I already paste whole index.ts file.
f
Try installing without Helm, in fact if I were you I’d avoid helm like the plague.
Also I am not sure this works
Copy code
'<http://alb.ingress.kubernetes.io/subnets|alb.ingress.kubernetes.io/subnets>': vpc.publicSubnetIds.join(',')
What type is vpc.publicSubnets, is it a promise?
l
Copy code
NAMESPACE           NAME                                              READY   STATUS    RESTARTS   AGE     IP             NODE                           NOMINATED NODE   READINESS GATES   LABELS
agromaster          agromaster-plot-5b554dc976-g2rmk                  1/1     Running   0          30h     10.0.188.247   ip-10-0-225-65.ec2.internal    <none>           <none>            name=agromaster-plot,pod-template-hash=5b554dc976
default             alb-aws-alb-ingress-controller-6dfc9fbf5d-c5twl   1/1     Running   0          5h23m   10.0.238.247   ip-10-0-225-65.ec2.internal    <none>           <none>            <http://app.kubernetes.io/instance=alb,app.kubernetes.io/name=aws-alb-ingress-controller,pod|app.kubernetes.io/instance=alb,app.kubernetes.io/name=aws-alb-ingress-controller,pod>
-template-hash=6dfc9fbf5d
kube-system         aws-node-4z6mm                                    1/1     Running   0          11d     10.0.225.65    ip-10-0-225-65.ec2.internal    <none>           <none>            controller-revision-hash=7bc84ccfc6,k8s-app=aws-node,pod-template-generation=2
kube-system         aws-node-ptgdv                                    1/1     Running   0          11d     10.0.104.149   ip-10-0-104-149.ec2.internal   <none>           <none>            controller-revision-hash=7bc84ccfc6,k8s-app=aws-node,pod-template-generation=2
kube-system         coredns-56678dcf76-cz275                          1/1     Running   0          11d     10.0.221.141   ip-10-0-225-65.ec2.internal    <none>           <none>            <http://eks.amazonaws.com/component=coredns,k8s-app=kube-dns,pod-template-hash=56678dcf76|eks.amazonaws.com/component=coredns,k8s-app=kube-dns,pod-template-hash=56678dcf76>
kube-system         coredns-56678dcf76-kw9dv                          1/1     Running   0          11d     10.0.205.16    ip-10-0-225-65.ec2.internal    <none>           <none>            <http://eks.amazonaws.com/component=coredns,k8s-app=kube-dns,pod-template-hash=56678dcf76|eks.amazonaws.com/component=coredns,k8s-app=kube-dns,pod-template-hash=56678dcf76>
kube-system         kube-proxy-q7q6b                                  1/1     Running   0          11d     10.0.104.149   ip-10-0-104-149.ec2.internal   <none>           <none>            controller-revision-hash=7f48466c4d,k8s-app=kube-proxy,pod-template-generation=1
kube-system         kube-proxy-xknvz                                  1/1     Running   0          11d     10.0.225.65    ip-10-0-225-65.ec2.internal    <none>           <none>            controller-revision-hash=7f48466c4d,k8s-app=kube-proxy,pod-template-generation=1
syngenta-platform   syngenta-platform-ndvi-6bf4f647c4-xn8z5           1/1     Running   0          30h     10.0.203.178   ip-10-0-225-65.ec2.internal    <none>           <none>            name=syngenta-platform-ndvi,pod-template-hash=6bf4f647c4
syngenta-platform   syngenta-platform-nginx-5f7647f8dd-jrlnj          1/1     Running   0          30h     10.0.125.119   ip-10-0-104-149.ec2.internal   <none>           <none>            name=syngenta-platform-nginx,pod-template-hash=5f7647f8dd
right now, it 
alb-ingress
is in default namespace. can it be a problem?
f
Please don’t code dump here
That’s related to my issues, working with Pulumi promises and needing them in kubernetes manifest yamls.
l
no, I don't have any yml file.
f
I didn’t mean actual yml files, but the manifest specs.
Your k8 pulumi objects are basically the “yaml files” in this context.
l
I don't have manifest spec too. may be, give me command. I only wrote index.ts to generate infra. not a single command to modify cluster. strictly on IaaC.
Open in Slack
PreviousNext
Powered by