https://pulumi.com logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
d
I don't think there's a way to do this natively in Pulumi. Resources are expected to be owned by a stack, and there doesn't seem to be any inverted dependency resources to add additional backends/urlmaps to an LB. If it's just for CloudRun, you could look into Serverless NEGs which support dynamic registration of services based on their name https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts
You could also make a dynamic provider to add additional path matchers to the urlmap with the gcp sdk for other cases, too. Worth upvoting this issue for support: https://github.com/hashicorp/terraform-provider-google/issues/8662
a
Hey Anthony thank you for the response. We are utilizing multiple NEGs with 2nd gen cloud functions (uses cloud run under the hood). Not seeing in the docs how / where I could accomplish adding multiple serverless NEGs to a single global external application load balancer https://cloud.google.com/api-gateway/docs/gateway-load-balancing
d
It's not well documented, and unfortunately I don't have access to the example I did at work anymore. Supported using URL Masks https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts#url_masks
a
what u posted is exactly what we are trying to do but but with pulumi. everytime we spin up a new service we need to add the NEG to the backend LB along with URLMap updates so we can route traffic based on headers etc…. IE I see we can accomplish this maybe via gcloud? https://cloud.google.com/load-balancing/docs/https/setting-up-query-and-header-routing#gcloud_6
common Anthony don’t leave me hanging GET that example!!!! 🥺
d
Alas, it's scratch code in a git repo I don't have access to anymore; and not local. Going to grab my laptop and do a quick example. Gcp LBs are very... Overloaded 😂
❤️ 1
a
excited
do you take paypal sir?
d
all my pulumi support is pro-bono. That way I'm not liable 😉
a
thinking
I thought about replacing the backend service by fetching the current one, adding the NEG for the service and doing that for every service but am getting a 409 The resource ‘projects/staging/global/backendServices/google-backend-service’ already exists, alreadyExists
Copy code
const backendService = new BackendService('google-backend-service', {
      project,
      loadBalancingScheme: 'EXTERNAL',
      name: 'google-backend-service',
      description: 'Backend service for custom domain API Gateway load balancers',
      logConfig: { enable: true },
      backends: [{
        group: neg.selfLink
      }]
    }, { parent: neg, deleteBeforeReplace: true, replaceOnChanges: ['*'] });
d
yep, pulumi expects full ownership of resources
so what you're looking for is:? • ForwardingRule + URLMap in the infra stack • CloudRun service + patch URLMap in other stacks
a
correct, for example we have two stacks. Google Stack: This is where we put global / foundational infra that can be referenced by other stacks / teams / services. The stack involves the following resources… • BackendService • URLMap • SSLCertificate • TargetHttpsProxy • GlobalForwardingRule Service Stack: This is where we create an API Gateway for the Cloud Run service and add it to the existing backend in the Google Stack. The stack involves he following resources… • Api • Config • Gateway • RegionNetworkEndpointGroup
trying to add the RegionNetworkEndpointGroup created in a child / dependent stack to the parent BackendService owned by the Google infra stack.
d
I think you'd create the BackendService for each service, and add it to the Urlmap path rules?
s
This is a real shot in the dark, and I don't have deep expertise in Google Cloud, but I think the resources you mentioned are used in this codebase from a workshop I did with @refined-pilot-45584: https://github.com/timbohiatt/gke-at-scale-pulumi
👀 1
a
was hoping just to create one Global Load Balancer for cost purposes
s
Hope that helps.
a
@dry-keyboard-94795 is it possible to modifying an existing UrlMap? I was running into the same issues as the backend service. I would do that as a compromise.
d
@stocky-restaurant-98004 correct; however in this case the services are created in a separate stack to the LB, where the LB is in the parent stack so doesn't have references straight away to the services
a
Would I run into the same issue if the UrlMap was in the parent stack?
d
however pulumi (+ Terraform) don't have this exposed as a resource
a
@dry-keyboard-94795 that was going to be my fallback but not sure how to run gcloud commands safely during CI/CD dep;loyment with pulumi
d
You could do it with a dynamic provider, or 
local.Command
a
I was trying to do a bit of research on defining a dynamic provider but was having trouble finding a good example close to my use case.
have not seen 
local.command
i can try and look this up real quick
s
Again, vague recollection on my part, but I do believe you have to use 
gcloud
with the Pulumi Command provider to do some of the stuff in Tim's repo because it's not supported in the provider.
👀 1
d
but not sure how to run gcloud commands safely during CI/CD dep;loyment with pulumi
this depends on how you have auth setup. A default setup where you've setup the application-default credentials should work, but gets a little more involved if you're using assumed roles
👀 1
s
@alert-midnight-11504 If you use TF, 
command.local.Command
in Pulumi often serves a similar purpose to 
null_provider
with a provisioner or whatever it's called.
a
I think local.command is the way to go.
s
Dropping down to use 
command.local.Command
to use the cloud provider's CLI is not uncommon. I've had to do it with all of the hyperscalers at one point or another.
👀 1
a
we are using application default, which is used by Google Cloud Build, and should be able to execute those commands if the service account has access.
d
a
Well… dang. But I think this is the cleanest route to go. Thank you all so much! and the repo you provided @stocky-restaurant-98004 is extremely helpful! Mind if keep yall posted on my progress? Going to try and implement this now.
👍 2
youre welcome 1
d
sure. I've also just gotten my LB up. sorry it took so long, had to setup a new gcp account 🤣
a
jeez
ya im a little worried on how little documentation there is around serverless NEGs and API Gateway at Google. Still only supports Swagger 2.0. Having the ability to add a managed gateway a Global External LB just went GA. 😬 🪦
d
the lack of docs was one of the reasons why we didn't go ahead with the project. + was in preview at the time
a
oh great…
trying not to cry
I don’t really have a choice, we have a service that is falling over, and need to split traffic off an already overloaded service / endpoint. The problem is we cannot just take the service down or have our provider hit us on a separate endpoint without the ability to rollback.
d
this is why I loved having Cloudflare sat infront of some of our services. Made it really easy to use Workers (ie, real code) to do network migrations
a
wait!
we are using cloudflare
go on…
d
it needs to be in DNS proxy mode. I think you're using GCP Managed certificates, so guessing you're not in proxy mode
a
we are using self managed ssl certificate from cloudflare 😛
we are in full strict SSL
IE I created an origin ssl cert.
how can i split traffic at the DNS by looking for a particular header?
d
if you're on enterprise, you can use an Origin Rule. Otherwise, you need to setup a worker on the hostname. Workers are effectively Express services
a
not on enterprise 😢. This is too freaking hilarious. The service we have in place today (also using express), tried splitting the traffic by doing a redirect / forward back to Cloudflare on a different route. But we kept getting 504 Gateway timeouts….
so i tried doing the splitting at the LB level, and now it sounds like doing at the DNS with a express service / worker in front of everything is the way to go?
I have no idea what im doing
d
it's a way to do it. Worth looking through their examples: https://developers.cloudflare.com/workers/examples/
👀 1
@alert-midnight-11504 do the services share the same hostname? It seems 
add-path-matcher
requires distinct hostnames. If all services are made in the same stack, this isn't a problem as the path matching rules can be combined
a
the service urls are different / dynamic when they are stood up / created
the dns subdomain is the same along with the path
I think we are okay on the path matcher meaning, one route goes to 
/webhooks/Authorization
on service A, and the other goes to 
/provider/Authorization
on service B
d
and all services share the same domain?
a
yes same subdomain
d
are they all made in the same stack?
a
to split the traffic one service uses a JWT Bearer Token and the other users a 
signature
header. Thats how we want to split the traffic using route rules
@dry-keyboard-94795 no they are in two seprate stacks. One stack per service infra.
d
looks like header matchers can't be added using the CLI without doing the full import, too
a
meaning I cannot use the GCloud CLI to add urlMaps like in this example? https://cloud.google.com/load-balancing/docs/https/setting-up-url-redirects-classic
d
You can update it like that, just increases the complexity of implementation
a
ill take anything at this point
d
if this is a temporary thing while you migrate, I'd be tempted to do it by hand
a
it is a temporary thing atm, but I was really hoping to have a strategy for bringing on new services from other stacks / teams…
IE we are going to be stuck doing this again once we wrap up another micro service implementation on the same api.company-domain.com
Whats the point of having a global load balancer for 1 service?
d
you could reference the service stacks in the infra stack, and update the urlmap during creation
a
I can’t because the service stacks depend on the google (base) stack. And if Create the LB after all the services are created, then I need to teardown / replace the global loadbalancer anytime a service changes 😬
d
the urlmap will update in place
a
^ what you mean? I might be missing something
d
to add a service, you only need to add it in the urlmap's route matcher
a
are you saying I can update 1 UrlMap reference and create multiple backends?
d
yes
the service stacks can create their own Backends + NEGs. so yes, would use multiple backend service resouces
a
Is this viable at scale? IE multiple Backend Services would mean a LB for each service i guess?
d
it'd still be just a single LB (ie, ForwardingRule)
a
ohhhhhhh
this diagram from google docs is tripping me up. Meaning I cannot find a way to actually implement this diagram
how do i go about updating the urlMap? using local command or is there a pulumi native way of doing this?
d
you can use Pulumi. Whenever a new service is added, you can update the code
a
IE would i 
UrlMap.get()
in the service stack to get a reference from base Google stack and use a 
transformation
?
d
do you create the URLMap in pulumi currently?
a
yes
but in google stack
s
This convo rules. I love watchin' y'all work through a real tough problem!
🤗 1
a
Hey @stocky-restaurant-98004 mind calling our managing investors and tell them what you just said. Thx! Met you guys at GoogleNext’23 I had nothing but great things to say, and got a free pair of Pulumi Socks ❤️
d
heh, is everyone copying M$oft with the socks now? I sitll have my vscode socks
🙌 1
a
Ya Aiven was handing out 🦀 socks too lol
d
so your google stack can contain an urlmap that looks something like this: https://gist.github.com/antdking/9adad9ec7d03d212be9de5f9159bd6cc Note that the backendservice IDs are loaded from the stack references
👀 1
hastily checks this actually works
a
this wont work because the service’s stack depend on the Base Google stack which is the code you are presenting.
Copy code
const servicesStack = new pulumi.StackReference(`${org}/gcp-lb-multistack-services/dev`);
// These could be sourced from multiple stacks
const s1BackendServiceId = servicesStack.requireOutput("be1Id");
const s2BackendServiceId = servicesStack.requireOutput("be2Id");
IE the services have not been created yet. they are created once the Google stack has been deployed 😞
d
the expectation is you update the code after the service stacks have initially run
a
I need to go the other way, where I am importing the url map and updating it
But this would create a dependency on the all the services being creating first.
d
correct
a
which means if one of those services changes, gets added, removed, I have to re-create the LB with the new state from the services
😬
d
it'll just update the urlmap. If you add this to the 
routeRules
, it'll not recreate everything:
Copy code
{
    service: s2BackendServiceId,
    priority: 3,
    matchRules: [
        {
            prefixMatch: "/s2",
        }
    ]
}
however as you say, the service stacks must have run before you update the code in your google stack
a
ya which, im thinking, would be another stack just to manage the LB deployment after all the services have deployed.
d
I think that is simpler than trying to update the urlmap inplace from the service stacks at this point
a
I might try going the urlMap update route first using the local.command CLI then if i get stuck ill try ur approach
d
additionally, the service stacks can just export their cloudrun IDs, and have the NEGs + BackendServices created in the google stack too
💯 1
👀 1
a
I am hoping the local command to update the URL map is just a few lines vs implementing and orchestrating another stack deployment life cycle.
d
there's a couple of limitations with `add-path-matcher`: • You can only have one pathMatcher per hostname • it doesn't support adding 
routeRules
, which are required for your header-based routing
a
😬
d
to do it with the CLI, you'll need to 
export
the urlmap config, update the yaml with additional path rules, then 
import
it. I'm not sure what protections there are if you say these stacks will run in parallel; though it should be fine as this is only for creates
a
ill be dead in the water if I cannot route based on headers
@dry-keyboard-94795 I think you approach is the cleanest and makes the most sense. Going to go heads down for a bit and see if i can get this implemented. Ill check-in in a little bit if thats alright. I def owe you a 🍺
Thank you so much for your pro-bono time 😛
d
no problem. It's an interesting problem, and I think would be a great feature to be added to the GCP provider in the future. AWS has had support for configuring this kind of routing decentralised since forever: https://www.pulumi.com/registry/packages/aws/api-docs/alb/listenerrule/
💯 1
👀 1
a
ya… last thing I want to do is be like.
d
For completeness, I've added the referenced services stack to that gist: https://gist.github.com/antdking/9adad9ec7d03d212be9de5f9159bd6cc
👀 1
a
☝️ @dry-keyboard-94795 this is amazing thank you!
s
@alert-midnight-11504 Hey Austin, @stocky-restaurant-98004 mentioned you might be interested in writing something up on this topic. I’d be happy to help support you, collaborate with you, etc., if you decide to go that route. Feel free to DM me.
🫡 1
a
@dry-keyboard-94795 Looks like your suggestion is working. Just finished implementation, now promoting to staging for additional testing, and then off to production. wish me luck 🙂
Hey @dry-keyboard-94795 @stocky-restaurant-98004 Just wanted to say thank you again. @dry-keyboard-94795’s suggested approach of just tacking on another stack to manage the Global LB and Url Matching rules downstream worked really well for us (the trade off being more orchestration using Pulumi which is not bad). Changes to the services just updates the LB UrlMaps instead of just redeploying the whole thing which was critical for us. We have successfully broken up our monolithic webhook service into smaller pieces. We ended up creating a dedicated card authorization service to handle just live card authorizations coming from the card networks. This has resulted in better tracing, logging, monitoring, sizing / scaling configurations. Pulumi Makes it extremely easy to navigate and reason about when making large infrastructure decisions like this. Without IaC and Pulumi’s more developer focused / friendly approach to deploying, testing, and managing changes to our infrastructure over time, this would be much more challenging. Deploying our infrastructure is only half the battle. Building, bundling, deploying, and orchestrating our services from source as part of that IaC process allows our developer’s, who are not necessarily platform / infra experts, to focus on their best work without having to think too much about higher level abstractions at the Cloud Native / GCP Project levels. I really want to do a write-up on this, but we are not quite out of the woods yet. Our initial load tests and new infrastructure as identified other key areas for us to focus our attention as we continue to scale.
❤️ 2
Just wanted to give a huge shoutout to @dry-keyboard-94795 and the Pulumi community for being so awesome, approachable, and helpful through these stressful times (for me anyways 😆)
d
Thanks, that's great to hear. I'm glad Pulumi has helped you, and hopefully GCP makes urlmaps a little more flexible in the future. We're always here to help
Going with IaC should also help with your compliance requirements. I know our assessors loved it when we said "we just click this button and magic happens" 😂 You'll likely be interested in the latest changes to CrossGuard down the road 
</mercilessUpselling>
🤣
14 Views
Open in Slack
PreviousNext
Powered by