https://pulumi.com logo
Join Slack
Powered by
<@UGGR2EYP3> you can give the cluster `nodeSubnetI...
# general
s
@busy-pizza-73563 you can give the cluster 
nodeSubnetIds
when you create it - if you put them in the same AZ the nodes will all end up there
b
What should those be, though?
s
The IDs of whatever VPC subnets you want the worker nodes to be in
b
Ok, so I should manually create a VPC first.
s
If you want to use the default VPC, you could also get the ID of the subnet in the zone you want to use
b
What if I don't want to use the default VPC, but the one generated by 
eks.Cluster()
?
s
Hmm, I don’t see a way to control zones there. @lemon-spoon-91807 may have a better answer though
b
I see no 
zone
option in 
aws.ec2.Vpc()
either, though.
l
I don't know enough to answer this confidently.
However, it looks like eks.Cluster has this:
Copy code
/**
     * Nested argument for the VPC associated with your cluster. Amazon EKS VPC resources have specific requirements to work properly with Kubernetes. For more information, see [Cluster VPC Considerations](<https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html>) and [Cluster Security Group Considerations](<https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html>) in the Amazon EKS User Guide. Configuration detailed below.
     */
    public readonly vpcConfig: pulumi.Output<{ securityGroupIds?: string[], subnetIds: string[], vpcId: string }>;
So it looks like we can obtain the vpcId and use that accordingly.
for example, you could then do: 
Vpc.fromExistingIds(..., { vpcId: eksCluster.vpcConfig.vpcId })
b
Thanks both! Unfortunately I still have no idea how to pin any of those resources to an AZ.
l
sorry, i think i'm not fully caught up
Ok. so you're trying to make an EKS cluster, but limit it to a particular AZ? is that correct?
b
Yes, because you can only mount EBS volumes to instances in the same AZ.
l
ok. so i'm not savvy about this area. @creamy-potato-29402 do you know how this shoudl be handled with EKS?
s
@lemon-spoon-91807 the way to do this is to create the worker nodes in the same subnet (which exists in one AZ only)
What’s unclear is whether the 
eks.Cluster
on it’s own can create a VPC with a single subnet in a specified zone
b
Right now the instances are spread throughout the whole region, so they can't all mount a specific EBS volume, in a specific AZ.
s
(There may be further restrictions - e.g. i’m not sure if EKS needs to be across multiple AZs)
l
I believe you 🙂 i just don't know how you control that for EKS. I'll need @creamy-potato-29402 to weign in.
note: it looks like when you make a cluster, you can provide VPC info
so, if you created a VPC, and asked it to only use one AZ, and then created the cluster from that, then maybe that would work?
s
Looks to me like the way to do it is to create the VPC you want, then use Vpc.FromExistingIDs as you say
b
Well, I didn't find how you can control them for `VPC`s, either.
s
Assuming that is compatible with EKS
@busy-pizza-73563 you can’t - VPC is a region-wide concept
l
well, if you create your own VPC, you don't really need to do VPC.fromExistingIds
you'd literally just "new up" the VPC with the config you'd like.
s
Ah true, you could just pass the VPC in I suppose
b
As I said above, there's no 
zone
config in 
aws.ec2.Vpc()
.
s
@busy-pizza-73563 there’s no such thing as a VPC which exists only in one AZ though, it’s the subnets which exist in a given AZ.
l
one of the args to VPC is `numberOfAvailabilityZones". So, in this case, you would just say 
numberOfAvailabilityZones=1
s
There is on 
aws.ec2.Subnet
though (it’s called 
availabilityZone
)
l
right (to what Jen said), 
numberOfAvailabilityZones
just affects how we partition all the subnets
so, if you ask for '1', we'll put all the subnets in the same AZ.
b
And how do I link a 
Subnet
to a 
Vpc
?
s
Yup - that’s likely the easiest way to do this if you don’t need it in a specific AZ (just a single one)
@busy-pizza-73563 with 
vpcId
on the subnet
l
we can make the subnet on your behalf
b
Damn, this whole AWS infrastructure is sooo confusing. 😐
l
indeed 🙂
let me step back a second
do you have existings VPCs/subnets you're trying to use?
or are you trying to create an appropriate VPC/subnet config to be used here?
b
No, I'm only using 
eks.Cluster()
now.
s
And a further question: do you have existing EBS volumes in a particular AZ that you want to be able to attach?
b
Copy code
const cluster = new eks.Cluster('cluster', {
  skipDefaultNodeGroup: true,
  ...
});
cluster.createNodeGroup('worker', {
...
});
l
No, I'm only using 
eks.Cluster()
now.
Ok. if you don't have an existing VPC you need to use, you can just create a new one, set 'numOfAvailAbilityZones=1' on it
say what kinds of subnets you want (public/private/isolated)
and we'll go configure that.
b
I can't find 
numOfAvailAbilityZones
in 
aws.ec2.Vpc()
.
l
then, it sounds like you can pass in the appropraite info from that Vpc to the Cluster when you create it as part of the vpcConfig property.
sorry, use awsx.ec2.Vpc 🙂
i mean, you can directly use raw aws.ec2.Vpc, it's just a lot more work
awsx.ec2.Vpc allows you to much more simply configure an entire Vpc with appropriate subnets, natgateways and internetgateways
b
Shouldn't that be an option to 
cluster.createNodeGroup()
, though?
(and / or 
new eks.Cluster()
)
l
sorry, i don't know what cluster.createNodeGroup is. can you link me?
Shouldn't that be an option to 
cluster.createNodeGroup()
, though?
could you be more specific. what do you mean by "that"?
b
Oh, sorry, I meant the 
numOfAvailAbilityZones
option.
s
It is an option by proxy: createNodeGroup does not create a network, it uses one it is passed - the nodeSubnetIds control which AZs they are in, because subnets are tied to an AZ
l
I see:
1. i don't know 
@pulumi/eks
very well. sorry
2. tagging @creamy-potato-29402 do absorb this feedback in case we should do things here in hte future to improve the interaction/integration of these resources.
b
@stocky-spoon-28903 To answer your earlier question, I don't have an EBS, I just want to be able to mount a certain EBS on all instances.
l
EKS is very very very much in progress 🙂
Do you feel like you have enough information from me at this point @busy-pizza-73563 and @stocky-spoon-28903?
b
Yeah, I was hoping I don't have to read all AWS documentation to figure out how everything works together... 😐
l
It is def our goal to make EKS much easier to use. But it's also very much a WIP.
b
Duly noted!
l
your scenairo seems like a top priority to nail
b
Now, any idea how I could pin the Vpc / Subnet to a certain AZ, and not just say "use only one AZ at random"? 🙂
l
to do that, unfortunately, you would need to likely go to the aws level
and just start manually making the resources yourself
so make the Vpc
then explicitly create the subnet, specifying the exact AZ you want for it
most people don't want/need this level of control, so the awsx.Vpc abstraction is intended for making a VPC that follows AWS recommended best practices, without making you have to worry about al lthat stuff
b
Ok, makes sense. I was somehow under the impression that you can do in Pulumi everything you can from the AWS Console.
l
(and potentially doing it badly, like not properly distributing subnets across azs well)
you can!
sorry!
by 'manually', i meant: write code directly at the 'pulumi/aws' layer
don't use the 'pulumi/awsx' helper abstracation
sorry for the confusion
b
Ah, ok. 🙂
l
pulumi/awsx just sits on top of pulumi/aws itself. and you can totally see what it's doing
yeah, i should not say "manual" 🙂
b
Yeah, I know, I actually already looked at what 
eks.Cluster()
is doing. 🙂
l
question, for my own edification, but why do you need to be explicit about which AZ to use?
this is a first i've heard about someone wanting to do that
b
Versus "just use one AZ at random"?
l
right
b
Suppose I already have an EBS in a specific AZ (from a previous cluster), and want the EKS instances in the same AZ, so they can mount it.
l
(note; it would be semi-random). in that we will always choose them in order. so if you pick 
=1
we'll just always pick teh first AZ for that region.
ahhhhhh
ok... fair enough
this is def an interesting case.
thanks for explainign it
b
I can also do a write-up (with the solution, too - when I get there) in an issue in... probably the eks repo?
l
Absolutely
Alex will likely drive this from here
as these are improvements in his area of focus
he's super busy, but he says he'll read through this today
b
Btw, there would also be nice to be able to specify additional security group rules to be added to the default ones generated by eks.
Like opening up the SSH port on the EKS node instances.
c
yes please do
b
First I have to get this up and running. 🙂
c
this is one of the major sticking points for EKS generally, and I’m not entirely sure how people do stuff like this without a tool like Pulumi.
I’m still reading through this thread, but I think I get the gist of the problem so far…
b
I'm kinda new to both Pulumi (played with it with kubernetes, and I found it very intuitive so far) and AWS.
c
Alright, well you should know that EKS is extremely tricky.
b
In one sentence: I want to be able to pin the EKS node instances to a specified AZ.
c
It is only marginally easier than installing kubernetes yourself.
got it
give me a second to just finish seeing what everyone said … 🙂
b
Yeah, I've just noticed. 🙂 It's ages behind GKE.
c
Ok I’ve read it all through
So, yes: unfortunately unlike AKS and GKE, you basically have to do all of what you’re describing by yourself.
@lemon-spoon-91807 I do think that being able to “pin” things to specific AZs is very useful, particularly for EBS. This is a pretty sharp edge of AWS generally — many people don’t realize, e.g., that an ASG is not guaranteed to boot instances in the same AZ (or region), which means that when you scale up, you could boot a bunch of instances that can’t even mount EBS. Ouch!
l
understood. note that our Vpc exposes the subnets it created
c
Do you have thoughts on whether this would be a good candidate for making it into awsx? It’s relatively high on my list.
l
so you could def say: use the az for this subnet
for bogdon's case, he has an existing Vpc (since he has that AZ+subnet where his resource already lives)
so he would basically just need to reference that afaict
c
@busy-pizza-73563 unfortunately, to your main question, at this point I do think you “just have to understand all of AWS” in order to do what you want. I wish we had a better answer, but I actually think this belongs in the AWSx package instead of the EKS package — otherwise EKS will become the transitive closure of all AWSx abstractions that it touches.
@lemon-spoon-91807 ah, right, you’re saying that bogdan already has a VPC, so he can just reference it. I think the “create this pinned to this az” is a useful thing, too, though…
sorry, lots of info in the thread, hard to keep track of it all.
l
yeah, and i might be wrong. but htat's what it sounds like to me... since he has existing stuff he wants to use
b
I don't already have a VPC, though. :-)
c
So that’s why I said you’ll have to “just understand all of AWS” and make it manually.
b
I started from scratch with eks, my only issue is EBS volume mounting.
c
I am hoping to convince @lemon-spoon-91807 that we shoudl make this use case a little easier in awsx though
l
makes sense ot me. though i don't think it would be super hard to do in aws already. Just make a VPC. Add a subnet in the AZ you want...
seems like it would just be two steps.
then reference that VPC from the eks cluster.
b
Is it expected for the EKS node groups to always upgrade to the latest AMI image?
I just got an update without changing anything, and it was because the AMI image was just updated by Amazon.
So, I ended up with something like this:
Copy code
export function getSubnetIdInAZ(cluster: eks.Cluster, az: string) {
  const subnetIds = cluster.eksCluster.vpcConfig.subnetIds;
  const index = subnetIds.apply(ids =>
    Promise.all(
      ids.map(id =>
        aws.ec2.getSubnet({ id }).then(subnet => subnet.availabilityZone)
      )
    ).then(azs => azs.indexOf(az))
  );
  return index.apply(index => subnetIds[index]);
}
l
That seems complex 🙂 don't want to use async/await?
b
And then just
Copy code
cluster.createNodeGroup('worker', {
  ...
  nodeSubnetIds: [getSubnetIdInAZ(cluster, 'eu-central-1c')],
}
I still think the AZ could be moved to an option in 
ClusterNodeGroupOptions
. 🙂
l
seems reasonable to me from my 50k foot view
b
Doesn't really look much more readable...
Copy code
export function getSubnetIdInAZ(cluster: eks.Cluster, az: string) {
  const { subnetIds } = cluster.eksCluster.vpcConfig;
  const index = subnetIds.apply(async ids => {
    const availabilityZones = await Promise.all(
      ids.map(async id => (await aws.ec2.getSubnet({ id })).availabilityZone)
    );
    return availabilityZones.indexOf(az);
  });
  return index.apply(index => subnetIds[index]);
}
Can you spot any obvious refactoring?
Got it down to
Copy code
export function getSubnetIdInAZ(cluster: eks.Cluster, az: string) {
  const { subnetIds } = cluster.eksCluster.vpcConfig;
  return subnetIds.apply(async ids => {
    const subnets = await Promise.all(ids.map(id => aws.ec2.getSubnet({ id })));
    const subnet = subnets.find(subnet => subnet.availabilityZone === az);
    if (!subnet) {
      throw new Error(`No subnet found in ${az} zone`);
    }
    return subnet.id;
  });
}
s
Haven’t read the entire thread here but what is the function aiming to do here?
b
Given an eks.Cluster and an AZ name (e.g. 'eu-central-1c`), it returns the subnet located in that AZ.
s
Ah ok
b
So I got it working, thank you for your time, everyone!
I'll add issues for the things I struggled with in the following days.
s
Sounds good - it might be that there are some missing features and/or examples and/or docs that can be added here
b
Open in Slack
PreviousNext
Powered by