https://pulumi.com logo
s

stocky-spoon-28903

03/27/2019, 5:42 PM
@busy-pizza-73563 you can give the cluster
nodeSubnetIds
when you create it - if you put them in the same AZ the nodes will all end up there
b

busy-pizza-73563

03/27/2019, 5:45 PM
What should those be, though?
s

stocky-spoon-28903

03/27/2019, 5:46 PM
The IDs of whatever VPC subnets you want the worker nodes to be in
b

busy-pizza-73563

03/27/2019, 5:46 PM
Ok, so I should manually create a VPC first.
s

stocky-spoon-28903

03/27/2019, 5:49 PM
If you want to use the default VPC, you could also get the ID of the subnet in the zone you want to use
b

busy-pizza-73563

03/27/2019, 6:04 PM
What if I don't want to use the default VPC, but the one generated by
eks.Cluster()
?
s

stocky-spoon-28903

03/27/2019, 6:05 PM
Hmm, I don’t see a way to control zones there. @lemon-spoon-91807 may have a better answer though
b

busy-pizza-73563

03/27/2019, 6:07 PM
I see no
zone
option in
aws.ec2.Vpc()
either, though.
l

lemon-spoon-91807

03/27/2019, 6:13 PM
I don't know enough to answer this confidently.
However, it looks like eks.Cluster has this:
Copy code
/**
     * Nested argument for the VPC associated with your cluster. Amazon EKS VPC resources have specific requirements to work properly with Kubernetes. For more information, see [Cluster VPC Considerations](<https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html>) and [Cluster Security Group Considerations](<https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html>) in the Amazon EKS User Guide. Configuration detailed below.
     */
    public readonly vpcConfig: pulumi.Output<{ securityGroupIds?: string[], subnetIds: string[], vpcId: string }>;
So it looks like we can obtain the vpcId and use that accordingly.
for example, you could then do:
Vpc.fromExistingIds(..., { vpcId: eksCluster.vpcConfig.vpcId })
b

busy-pizza-73563

03/27/2019, 6:18 PM
Thanks both! Unfortunately I still have no idea how to pin any of those resources to an AZ.
l

lemon-spoon-91807

03/27/2019, 6:19 PM
sorry, i think i'm not fully caught up
Ok. so you're trying to make an EKS cluster, but limit it to a particular AZ? is that correct?
b

busy-pizza-73563

03/27/2019, 6:23 PM
Yes, because you can only mount EBS volumes to instances in the same AZ.
l

lemon-spoon-91807

03/27/2019, 6:23 PM
ok. so i'm not savvy about this area. @creamy-potato-29402 do you know how this shoudl be handled with EKS?
s

stocky-spoon-28903

03/27/2019, 6:24 PM
@lemon-spoon-91807 the way to do this is to create the worker nodes in the same subnet (which exists in one AZ only)
What’s unclear is whether the
eks.Cluster
on it’s own can create a VPC with a single subnet in a specified zone
b

busy-pizza-73563

03/27/2019, 6:24 PM
Right now the instances are spread throughout the whole region, so they can't all mount a specific EBS volume, in a specific AZ.
s

stocky-spoon-28903

03/27/2019, 6:24 PM
(There may be further restrictions - e.g. i’m not sure if EKS needs to be across multiple AZs)
l

lemon-spoon-91807

03/27/2019, 6:25 PM
I believe you 🙂 i just don't know how you control that for EKS. I'll need @creamy-potato-29402 to weign in.
note: it looks like when you make a cluster, you can provide VPC info
so, if you created a VPC, and asked it to only use one AZ, and then created the cluster from that, then maybe that would work?
s

stocky-spoon-28903

03/27/2019, 6:25 PM
Looks to me like the way to do it is to create the VPC you want, then use Vpc.FromExistingIDs as you say
b

busy-pizza-73563

03/27/2019, 6:25 PM
Well, I didn't find how you can control them for `VPC`s, either.
s

stocky-spoon-28903

03/27/2019, 6:26 PM
Assuming that is compatible with EKS
@busy-pizza-73563 you can’t - VPC is a region-wide concept
l

lemon-spoon-91807

03/27/2019, 6:26 PM
well, if you create your own VPC, you don't really need to do VPC.fromExistingIds
you'd literally just "new up" the VPC with the config you'd like.
s

stocky-spoon-28903

03/27/2019, 6:26 PM
Ah true, you could just pass the VPC in I suppose
b

busy-pizza-73563

03/27/2019, 6:26 PM
As I said above, there's no
zone
config in
aws.ec2.Vpc()
.
s

stocky-spoon-28903

03/27/2019, 6:26 PM
@busy-pizza-73563 there’s no such thing as a VPC which exists only in one AZ though, it’s the subnets which exist in a given AZ.
l

lemon-spoon-91807

03/27/2019, 6:26 PM
one of the args to VPC is `numberOfAvailabilityZones". So, in this case, you would just say
numberOfAvailabilityZones=1
s

stocky-spoon-28903

03/27/2019, 6:27 PM
There is on
aws.ec2.Subnet
though (it’s called
availabilityZone
)
l

lemon-spoon-91807

03/27/2019, 6:27 PM
right (to what Jen said),
numberOfAvailabilityZones
just affects how we partition all the subnets
so, if you ask for '1', we'll put all the subnets in the same AZ.
b

busy-pizza-73563

03/27/2019, 6:27 PM
And how do I link a
Subnet
to a
Vpc
?
s

stocky-spoon-28903

03/27/2019, 6:27 PM
Yup - that’s likely the easiest way to do this if you don’t need it in a specific AZ (just a single one)
@busy-pizza-73563 with
vpcId
on the subnet
l

lemon-spoon-91807

03/27/2019, 6:28 PM
we can make the subnet on your behalf
b

busy-pizza-73563

03/27/2019, 6:28 PM
Damn, this whole AWS infrastructure is sooo confusing. 😐
l

lemon-spoon-91807

03/27/2019, 6:28 PM
indeed 🙂
let me step back a second
do you have existings VPCs/subnets you're trying to use?
or are you trying to create an appropriate VPC/subnet config to be used here?
b

busy-pizza-73563

03/27/2019, 6:29 PM
No, I'm only using
eks.Cluster()
now.
s

stocky-spoon-28903

03/27/2019, 6:29 PM
And a further question: do you have existing EBS volumes in a particular AZ that you want to be able to attach?
b

busy-pizza-73563

03/27/2019, 6:29 PM
Copy code
const cluster = new eks.Cluster('cluster', {
  skipDefaultNodeGroup: true,
  ...
});
cluster.createNodeGroup('worker', {
...
});
l

lemon-spoon-91807

03/27/2019, 6:29 PM
No, I'm only using
eks.Cluster()
now.
Ok. if you don't have an existing VPC you need to use, you can just create a new one, set 'numOfAvailAbilityZones=1' on it
say what kinds of subnets you want (public/private/isolated)
and we'll go configure that.
b

busy-pizza-73563

03/27/2019, 6:30 PM
I can't find
numOfAvailAbilityZones
in
aws.ec2.Vpc()
.
l

lemon-spoon-91807

03/27/2019, 6:30 PM
then, it sounds like you can pass in the appropraite info from that Vpc to the Cluster when you create it as part of the vpcConfig property.
sorry, use awsx.ec2.Vpc 🙂
i mean, you can directly use raw aws.ec2.Vpc, it's just a lot more work
awsx.ec2.Vpc allows you to much more simply configure an entire Vpc with appropriate subnets, natgateways and internetgateways
b

busy-pizza-73563

03/27/2019, 6:32 PM
Shouldn't that be an option to
cluster.createNodeGroup()
, though?
(and / or
new eks.Cluster()
)
l

lemon-spoon-91807

03/27/2019, 6:32 PM
sorry, i don't know what cluster.createNodeGroup is. can you link me?
Shouldn't that be an option to
cluster.createNodeGroup()
, though?
could you be more specific. what do you mean by "that"?
Oh, sorry, I meant the
numOfAvailAbilityZones
option.
s

stocky-spoon-28903

03/27/2019, 6:34 PM
It is an option by proxy: createNodeGroup does not create a network, it uses one it is passed - the nodeSubnetIds control which AZs they are in, because subnets are tied to an AZ
l

lemon-spoon-91807

03/27/2019, 6:35 PM
I see:
1. i don't know
@pulumi/eks
very well. sorry
2. tagging @creamy-potato-29402 do absorb this feedback in case we should do things here in hte future to improve the interaction/integration of these resources.
b

busy-pizza-73563

03/27/2019, 6:35 PM
@stocky-spoon-28903 To answer your earlier question, I don't have an EBS, I just want to be able to mount a certain EBS on all instances.
l

lemon-spoon-91807

03/27/2019, 6:35 PM
EKS is very very very much in progress 🙂
Do you feel like you have enough information from me at this point @busy-pizza-73563 and @stocky-spoon-28903?
b

busy-pizza-73563

03/27/2019, 6:37 PM
Yeah, I was hoping I don't have to read all AWS documentation to figure out how everything works together... 😐
l

lemon-spoon-91807

03/27/2019, 6:37 PM
It is def our goal to make EKS much easier to use. But it's also very much a WIP.
b

busy-pizza-73563

03/27/2019, 6:38 PM
Duly noted!
l

lemon-spoon-91807

03/27/2019, 6:39 PM
your scenairo seems like a top priority to nail
b

busy-pizza-73563

03/27/2019, 6:39 PM
Now, any idea how I could pin the Vpc / Subnet to a certain AZ, and not just say "use only one AZ at random"? 🙂
l

lemon-spoon-91807

03/27/2019, 6:39 PM
to do that, unfortunately, you would need to likely go to the aws level
and just start manually making the resources yourself
so make the Vpc
then explicitly create the subnet, specifying the exact AZ you want for it
most people don't want/need this level of control, so the awsx.Vpc abstraction is intended for making a VPC that follows AWS recommended best practices, without making you have to worry about al lthat stuff
b

busy-pizza-73563

03/27/2019, 6:41 PM
Ok, makes sense. I was somehow under the impression that you can do in Pulumi everything you can from the AWS Console.
l

lemon-spoon-91807

03/27/2019, 6:41 PM
(and potentially doing it badly, like not properly distributing subnets across azs well)
you can!
sorry!
by 'manually', i meant: write code directly at the 'pulumi/aws' layer
don't use the 'pulumi/awsx' helper abstracation
sorry for the confusion
b

busy-pizza-73563

03/27/2019, 6:41 PM
Ah, ok. 🙂
l

lemon-spoon-91807

03/27/2019, 6:42 PM
pulumi/awsx just sits on top of pulumi/aws itself. and you can totally see what it's doing
yeah, i should not say "manual" 🙂
b

busy-pizza-73563

03/27/2019, 6:42 PM
Yeah, I know, I actually already looked at what
eks.Cluster()
is doing. 🙂
l

lemon-spoon-91807

03/27/2019, 6:42 PM
question, for my own edification, but why do you need to be explicit about which AZ to use?
this is a first i've heard about someone wanting to do that
b

busy-pizza-73563

03/27/2019, 6:43 PM
Versus "just use one AZ at random"?
l

lemon-spoon-91807

03/27/2019, 6:43 PM
right
b

busy-pizza-73563

03/27/2019, 6:44 PM
Suppose I already have an EBS in a specific AZ (from a previous cluster), and want the EKS instances in the same AZ, so they can mount it.
l

lemon-spoon-91807

03/27/2019, 6:44 PM
(note; it would be semi-random). in that we will always choose them in order. so if you pick
=1
we'll just always pick teh first AZ for that region.
ahhhhhh
ok... fair enough
this is def an interesting case.
thanks for explainign it
b

busy-pizza-73563

03/27/2019, 6:44 PM
I can also do a write-up (with the solution, too - when I get there) in an issue in... probably the eks repo?
l

lemon-spoon-91807

03/27/2019, 6:45 PM
Absolutely
Alex will likely drive this from here
as these are improvements in his area of focus
he's super busy, but he says he'll read through this today
b

busy-pizza-73563

03/27/2019, 6:46 PM
Btw, there would also be nice to be able to specify additional security group rules to be added to the default ones generated by eks.
Like opening up the SSH port on the EKS node instances.
c

creamy-potato-29402

03/27/2019, 6:47 PM
yes please do
b

busy-pizza-73563

03/27/2019, 6:47 PM
First I have to get this up and running. 🙂
c

creamy-potato-29402

03/27/2019, 6:47 PM
this is one of the major sticking points for EKS generally, and I’m not entirely sure how people do stuff like this without a tool like Pulumi.
I’m still reading through this thread, but I think I get the gist of the problem so far…
b

busy-pizza-73563

03/27/2019, 6:48 PM
I'm kinda new to both Pulumi (played with it with kubernetes, and I found it very intuitive so far) and AWS.
c

creamy-potato-29402

03/27/2019, 6:49 PM
Alright, well you should know that EKS is extremely tricky.
b

busy-pizza-73563

03/27/2019, 6:49 PM
In one sentence: I want to be able to pin the EKS node instances to a specified AZ.
c

creamy-potato-29402

03/27/2019, 6:49 PM
It is only marginally easier than installing kubernetes yourself.
got it
give me a second to just finish seeing what everyone said … 🙂
b

busy-pizza-73563

03/27/2019, 6:49 PM
Yeah, I've just noticed. 🙂 It's ages behind GKE.
c

creamy-potato-29402

03/27/2019, 6:53 PM
Ok I’ve read it all through
So, yes: unfortunately unlike AKS and GKE, you basically have to do all of what you’re describing by yourself.
@lemon-spoon-91807 I do think that being able to “pin” things to specific AZs is very useful, particularly for EBS. This is a pretty sharp edge of AWS generally — many people don’t realize, e.g., that an ASG is not guaranteed to boot instances in the same AZ (or region), which means that when you scale up, you could boot a bunch of instances that can’t even mount EBS. Ouch!
l

lemon-spoon-91807

03/27/2019, 6:56 PM
understood. note that our Vpc exposes the subnets it created
c

creamy-potato-29402

03/27/2019, 6:56 PM
Do you have thoughts on whether this would be a good candidate for making it into awsx? It’s relatively high on my list.
l

lemon-spoon-91807

03/27/2019, 6:56 PM
so you could def say: use the az for this subnet
for bogdon's case, he has an existing Vpc (since he has that AZ+subnet where his resource already lives)
so he would basically just need to reference that afaict
c

creamy-potato-29402

03/27/2019, 6:57 PM
@busy-pizza-73563 unfortunately, to your main question, at this point I do think you “just have to understand all of AWS” in order to do what you want. I wish we had a better answer, but I actually think this belongs in the AWSx package instead of the EKS package — otherwise EKS will become the transitive closure of all AWSx abstractions that it touches.
@lemon-spoon-91807 ah, right, you’re saying that bogdan already has a VPC, so he can just reference it. I think the “create this pinned to this az” is a useful thing, too, though…
sorry, lots of info in the thread, hard to keep track of it all.
l

lemon-spoon-91807

03/27/2019, 6:59 PM
yeah, and i might be wrong. but htat's what it sounds like to me... since he has existing stuff he wants to use
b

busy-pizza-73563

03/27/2019, 7:24 PM
I don't already have a VPC, though. :-)
c

creamy-potato-29402

03/27/2019, 7:24 PM
So that’s why I said you’ll have to “just understand all of AWS” and make it manually.
b

busy-pizza-73563

03/27/2019, 7:25 PM
I started from scratch with eks, my only issue is EBS volume mounting.
c

creamy-potato-29402

03/27/2019, 7:25 PM
I am hoping to convince @lemon-spoon-91807 that we shoudl make this use case a little easier in awsx though
l

lemon-spoon-91807

03/27/2019, 7:25 PM
makes sense ot me. though i don't think it would be super hard to do in aws already. Just make a VPC. Add a subnet in the AZ you want...
seems like it would just be two steps.
then reference that VPC from the eks cluster.
b

busy-pizza-73563

03/27/2019, 11:09 PM
Is it expected for the EKS node groups to always upgrade to the latest AMI image?
I just got an update without changing anything, and it was because the AMI image was just updated by Amazon.
So, I ended up with something like this:
Copy code
export function getSubnetIdInAZ(cluster: eks.Cluster, az: string) {
  const subnetIds = cluster.eksCluster.vpcConfig.subnetIds;
  const index = subnetIds.apply(ids =>
    Promise.all(
      ids.map(id =>
        aws.ec2.getSubnet({ id }).then(subnet => subnet.availabilityZone)
      )
    ).then(azs => azs.indexOf(az))
  );
  return index.apply(index => subnetIds[index]);
}
l

lemon-spoon-91807

03/27/2019, 11:14 PM
That seems complex 🙂 don't want to use async/await?
b

busy-pizza-73563

03/27/2019, 11:16 PM
And then just
Copy code
cluster.createNodeGroup('worker', {
  ...
  nodeSubnetIds: [getSubnetIdInAZ(cluster, 'eu-central-1c')],
}
I still think the AZ could be moved to an option in
ClusterNodeGroupOptions
. 🙂
l

lemon-spoon-91807

03/27/2019, 11:18 PM
seems reasonable to me from my 50k foot view
b

busy-pizza-73563

03/27/2019, 11:46 PM
Doesn't really look much more readable...
Copy code
export function getSubnetIdInAZ(cluster: eks.Cluster, az: string) {
  const { subnetIds } = cluster.eksCluster.vpcConfig;
  const index = subnetIds.apply(async ids => {
    const availabilityZones = await Promise.all(
      ids.map(async id => (await aws.ec2.getSubnet({ id })).availabilityZone)
    );
    return availabilityZones.indexOf(az);
  });
  return index.apply(index => subnetIds[index]);
}
Can you spot any obvious refactoring?
Got it down to
Copy code
export function getSubnetIdInAZ(cluster: eks.Cluster, az: string) {
  const { subnetIds } = cluster.eksCluster.vpcConfig;
  return subnetIds.apply(async ids => {
    const subnets = await Promise.all(ids.map(id => aws.ec2.getSubnet({ id })));
    const subnet = subnets.find(subnet => subnet.availabilityZone === az);
    if (!subnet) {
      throw new Error(`No subnet found in ${az} zone`);
    }
    return subnet.id;
  });
}
s

stocky-spoon-28903

03/28/2019, 12:11 AM
Haven’t read the entire thread here but what is the function aiming to do here?
b

busy-pizza-73563

03/28/2019, 12:13 AM
Given an eks.Cluster and an AZ name (e.g. 'eu-central-1c`), it returns the subnet located in that AZ.
s

stocky-spoon-28903

03/28/2019, 12:14 AM
Ah ok
b

busy-pizza-73563

03/28/2019, 12:35 AM
So I got it working, thank you for your time, everyone!
I'll add issues for the things I struggled with in the following days.
s

stocky-spoon-28903

03/28/2019, 12:42 AM
Sounds good - it might be that there are some missing features and/or examples and/or docs that can be added here
b

busy-pizza-73563

04/01/2019, 2:12 PM