https://pulumi.com logo
Title
e

elegant-activity-51782

03/21/2023, 1:08 PM
Hey folks. I am trying to update an AWSx VPC to have one public Subnet, one private subnet then a final subnet in each VPC that will be responsible for attaching to TGWs. has anyone done this before?
s

stocky-restaurant-98004

03/21/2023, 2:55 PM
Yes! Well, sorta. Pretty close. Check it out: https://github.com/pulumi/workshops/tree/main/aws-advanced-networking
I also authored the initial version of the VPC component FWIW. Most major modifications of the subnets aren't really possible because the CIDR block calculation gets really difficult. It's an unfortunate limitation.
e

elegant-activity-51782

03/21/2023, 2:57 PM
Isolated Subnets for TGW attachments work?
s

stocky-restaurant-98004

03/21/2023, 2:57 PM
Also worth noting just in case you weren't aware: You should split your VPCs across at least 3 AZs in a production-ready architecture.
e

elegant-activity-51782

03/21/2023, 2:57 PM
Yes totally aware of that one 😛
s

stocky-restaurant-98004

03/21/2023, 2:57 PM
Yes. They're "isolated" in the sense that they do not have a route to a NAT Gateway.
But there's nothing that says they can't have a route to a TGW if you add it. (There's also nothing that says you can't add a route to an isolated subnet to a NAT Gateway after the fact - they're only created "isolated".)
e

elegant-activity-51782

03/21/2023, 3:00 PM
And then once I have the TGW attachment attached to isolated VPC Subnets I’d just need to add the Private subnets to the TGW routes?
s

stocky-restaurant-98004

03/21/2023, 3:00 PM
You're also probably aware, but if you want to save money, since you have the TGW, you can route egress traffic through a hub VPC and you don't need to have NAT Gateways on each spoke. (That's what we do in the codebase I linked.)
Add routes to the private subnets so that 0.0.0.0/0 goes to the TGW (assuming that you're implementing centralized egress like I described above).
e

elegant-activity-51782

03/21/2023, 3:01 PM
I am totally in a POC stage and am not a “networking” guy at all so this is why I am doing bunch of fun disccovery stuff
s

stocky-restaurant-98004

03/21/2023, 3:02 PM
What traffic is going through the TGW?
Like, traffic to other VPCs, traffic to the internet, or both?
e

elegant-activity-51782

03/21/2023, 3:03 PM
Probably traffic to other VPCs. Although does the comment you made
since you have the TGW, you can route egress traffic through a hub VPC and you don't need to have NAT Gateways on each spoke. (That's what we do in the codebase I linked.)
also mean I could just do internet traffic through TGW to hub and save money that way?
s

stocky-restaurant-98004

03/21/2023, 3:05 PM
Yes. Exactly. There's 2 reasons why you'd want centralized egress (routing all traffic to the internet through a hub VPC): 1. You don't have to pay for NAT Gateways for each VPC, which get expensive at scale. 2. You can throw a firewall in that hub VPC (also contained in that codebase I linked) and also have centralized inspection (and only 1 set of rules to maintain) because all traffic flows through a single point.
e

elegant-activity-51782

03/21/2023, 3:06 PM
Don’t you end up just eating any savings in the TGW costs for egressing out internet traffic?
s

stocky-restaurant-98004

03/21/2023, 3:07 PM
I don't think so, but I never actually checked the costs...
e

elegant-activity-51782

03/21/2023, 3:08 PM
because according to https://aws.amazon.com/transit-gateway/pricing/ you pay per gb of data transfered through
s

stocky-restaurant-98004

03/21/2023, 3:09 PM
TGW is 2 cents per GB. NAT Gateway is 4.5 cents per hour, plus 4.5 cents per GB, so unless I'm missing something, that's a clear cost savings.
(I am definitely not an expert on cost savings.)
e

elegant-activity-51782

03/21/2023, 3:10 PM
Well I guess you’d go Spoke VPC -> TGW -> Hub VPC -> Public Subnet -> IGW correct?
Therefor ignoring NAT Gateway altogether?
s

stocky-restaurant-98004

03/21/2023, 3:11 PM
Still has to go out a NAT Gateway, I believe.
e

elegant-activity-51782

03/21/2023, 3:12 PM
so then you’d still eat the NAT Gateway costs in Hub VPCC
s

stocky-restaurant-98004

03/21/2023, 3:12 PM
But I think once you hit a couple of spokes, you're saving money. NAT Gateways tend to be expensive enough that people still roll their own. You typically want 1 gateway per AZ, per VPC, so that gets pricey quick. I believe NAT gateways are the biggest cost in running a VPC.
<- Again, no expert. YMMV.
I do know that you want to avoid cross-AZ traffic, so do be careful when you add your routes, no matter what architecture you decide on.
(Avoid cross-AZ traffic because of costs)
e

elegant-activity-51782

03/21/2023, 3:14 PM
Yeah I know for sure we want to go with a Hub-Spoke model. Just not sure in practicality how that actually shakes out
s

stocky-restaurant-98004

03/21/2023, 3:16 PM
How many spokes do you have?
e

elegant-activity-51782

03/21/2023, 3:16 PM
like I said very much in a POC/Discovery phase now. I can imagine we could scale for things my team manages to 10-15+?
s

stocky-restaurant-98004

03/21/2023, 3:17 PM
My intuition (and again, run them numbers - don't take my word for it) is that you'll see significant cost savings with TGW and centralized egress.
I developed that codebase for a workshop I did with some AWS networking SAs. I think they hang out here (and they're also probably in this channel): https://cloud-network-as-code.slack.com There's people in that Slack that will definitely be able to give you a second opinion, but I think you're on the right track with TGW and centralized egress.
Yeah, they here. @miniature-rocket-28706 and @miniature-laptop-27301 should be able to validate that you'll see significant cost savings with TGW and centralized egress for 10 to 15 VPCs (assuming they're not like... going across regions or anything wild).
@elegant-activity-51782 You might also want to check out Tailscale for a VPN solution if you have a need. Their product is really, really nice. Uses UDP and NAT to remove the need for poking holes in firewalls. https://tailscale.com/
m

miniature-laptop-27301

03/22/2023, 7:04 AM
Hi @stocky-restaurant-98004, Yes, me and Andy are in the cloud-network-as-code slack also. Regarding, the centralised egress, there are some caveats like traffic amount that affect the cost but in general that pattern is cheaper.