Channels
pulumi-ai
package-authoring
pulumi-cdk
oracle-cloud-infrastructure
learn-pulumi-events
linen
registry
built-with-pulumi
pulumi-cloud
contribex
testingtesting321
hacktoberfest
cloudengineering
yaml
pulumi-crosscode
content-share
finops
multi-language-hackathon
office-hours
workshops
blog-posts
localstack
welcome
gitlab
pulumi-kubernetes-operator
jobs
aws
pulumi-deployments
dotnet
golang
announcements
java
pulumiverse
python
install
getting-started
cloudengineering-support
testingtesting123
hackathon-03-19-2020
typescript
google-cloud
contribute
azure
kubernetes
docs
automation-api
status
general
Title
e
elegant-activity-51782
03/21/2023, 1:08 PMHey folks. I am trying to update an AWSx VPC to have one public Subnet, one private subnet then a final subnet in each VPC that will be responsible for attaching to TGWs. has anyone done this before?
s
stocky-restaurant-98004
03/21/2023, 2:55 PMYes! Well, sorta. Pretty close. Check it out: https://github.com/pulumi/workshops/tree/main/aws-advanced-networking
I also authored the initial version of the VPC component FWIW. Most major modifications of the subnets aren't really possible because the CIDR block calculation gets really difficult. It's an unfortunate limitation.
e
elegant-activity-51782
03/21/2023, 2:57 PMIsolated Subnets for TGW attachments work?
s
stocky-restaurant-98004
03/21/2023, 2:57 PMAlso worth noting just in case you weren't aware: You should split your VPCs across at least 3 AZs in a production-ready architecture.
e
elegant-activity-51782
03/21/2023, 2:57 PMYes totally aware of that one đ
s
stocky-restaurant-98004
03/21/2023, 2:57 PMYes. They're "isolated" in the sense that they do not have a route to a NAT Gateway.
But there's nothing that says they can't have a route to a TGW if you add it. (There's also nothing that says you can't add a route to an isolated subnet to a NAT Gateway after the fact - they're only created "isolated".)
Specifically, you'll want to check this file out: https://github.com/pulumi/workshops/blob/main/aws-advanced-networking/spoke.py
e
elegant-activity-51782
03/21/2023, 3:00 PMAnd then once I have the TGW attachment attached to isolated VPC Subnets Iâd just need to add the Private subnets to the TGW routes?
s
stocky-restaurant-98004
03/21/2023, 3:00 PMYou're also probably aware, but if you want to save money, since you have the TGW, you can route egress traffic through a hub VPC and you don't need to have NAT Gateways on each spoke. (That's what we do in the codebase I linked.)
Add routes to the private subnets so that 0.0.0.0/0 goes to the TGW (assuming that you're implementing centralized egress like I described above).
e
elegant-activity-51782
03/21/2023, 3:01 PMI am totally in a POC stage and am not a ânetworkingâ guy at all so this is why I am doing bunch of fun disccovery stuff
s
stocky-restaurant-98004
03/21/2023, 3:02 PMWhat traffic is going through the TGW?
Like, traffic to other VPCs, traffic to the internet, or both?
e
elegant-activity-51782
03/21/2023, 3:03 PMProbably traffic to other VPCs.
Although does the comment you made
since you have the TGW, you can route egress traffic through a hub VPC and you don't need to have NAT Gateways on each spoke. (That's what we do in the codebase I linked.)s
stocky-restaurant-98004
03/21/2023, 3:05 PMYes. Exactly. There's 2 reasons why you'd want centralized egress (routing all traffic to the internet through a hub VPC):
1. You don't have to pay for NAT Gateways for each VPC, which get expensive at scale.
2. You can throw a firewall in that hub VPC (also contained in that codebase I linked) and also have centralized inspection (and only 1 set of rules to maintain) because all traffic flows through a single point.
e
elegant-activity-51782
03/21/2023, 3:06 PMDonât you end up just eating any savings in the TGW costs for egressing out internet traffic?
s
stocky-restaurant-98004
03/21/2023, 3:07 PMI don't think so, but I never actually checked the costs...
e
elegant-activity-51782
03/21/2023, 3:08 PMbecause according to https://aws.amazon.com/transit-gateway/pricing/ you pay per gb of data transfered through
s
stocky-restaurant-98004
03/21/2023, 3:09 PMTGW is 2 cents per GB. NAT Gateway is 4.5 cents per hour, plus 4.5 cents per GB, so unless I'm missing something, that's a clear cost savings.
(I am definitely not an expert on cost savings.)
e
elegant-activity-51782
03/21/2023, 3:10 PMWell I guess youâd go Spoke VPC -> TGW -> Hub VPC -> Public Subnet -> IGW correct?
Therefor ignoring NAT Gateway altogether?
s
stocky-restaurant-98004
03/21/2023, 3:11 PMStill has to go out a NAT Gateway, I believe.
e
elegant-activity-51782
03/21/2023, 3:12 PMso then youâd still eat the NAT Gateway costs in Hub VPCC
s
stocky-restaurant-98004
03/21/2023, 3:12 PMBut I think once you hit a couple of spokes, you're saving money. NAT Gateways tend to be expensive enough that people still roll their own. You typically want 1 gateway per AZ, per VPC, so that gets pricey quick. I believe NAT gateways are the biggest cost in running a VPC.
<- Again, no expert. YMMV.
I do know that you want to avoid cross-AZ traffic, so do be careful when you add your routes, no matter what architecture you decide on.
(Avoid cross-AZ traffic because of costs)
e
elegant-activity-51782
03/21/2023, 3:14 PMYeah I know for sure we want to go with a Hub-Spoke model. Just not sure in practicality how that actually shakes out
s
stocky-restaurant-98004
03/21/2023, 3:16 PMHow many spokes do you have?
e
elegant-activity-51782
03/21/2023, 3:16 PMlike I said very much in a POC/Discovery phase now. I can imagine we could scale for things my team manages to 10-15+?
s
stocky-restaurant-98004
03/21/2023, 3:17 PMMy intuition (and again, run them numbers - don't take my word for it) is that you'll see significant cost savings with TGW and centralized egress.
I developed that codebase for a workshop I did with some AWS networking SAs. I think they hang out here (and they're also probably in this channel): https://cloud-network-as-code.slack.com
There's people in that Slack that will definitely be able to give you a second opinion, but I think you're on the right track with TGW and centralized egress.
Yeah, they here. @miniature-rocket-28706 and @miniature-laptop-27301 should be able to validate that you'll see significant cost savings with TGW and centralized egress for 10 to 15 VPCs (assuming they're not like... going across regions or anything wild).
@elegant-activity-51782 You might also want to check out Tailscale for a VPN solution if you have a need. Their product is really, really nice. Uses UDP and NAT to remove the need for poking holes in firewalls. https://tailscale.com/
m
miniature-laptop-27301
03/22/2023, 7:04 AMHi @stocky-restaurant-98004,
Yes, me and Andy are in the cloud-network-as-code slack also.
Regarding, the centralised egress, there are some caveats like traffic amount that affect the cost but in general that pattern is cheaper.