https://pulumi.com logo
Powered by
#general
Title
# general
a

alert-midnight-11504

10/25/2023, 3:17 PM
Hey all got myself in a bit of a pickle. Is there a way to modify a resource such as Pulumi’s GCP Classic BackendService to add / remove NEGs (Network Endpoint Group) after the base resource has been created? The problem I am having is the backend is created in another stack 
google
to manage our base / global infrastructure and is deployed before our other 
service
stacks. The backend service depends on a the NEG being defined (even though its optional) when the backend is being created. The NEG urls are dynamic and not static (Cloud Run Service URL). I am not sure how I am suppose to have 1 backend for many services, especially if each service is created in parallel. If I create all the services before creating the backend service (load balancer) that means I would have to tear down the entire backend every time a service changes which is not ideal. Thoughts? IE trying to do something like this…
Copy code
BackendService.get(`${namespace}-url-map`, globalExternalLoadBalancer.backendSelfLink, {}, { transformations: [(args) => {
      return {
        ...args,
        props: {
          ...args.props,
          backends: (args.props.backends || []).push({
            group: neg.selfLink
          })
        }
      }
    }]});
d

dry-keyboard-94795

10/25/2023, 4:15 PM
I don't think there's a way to do this natively in Pulumi. Resources are expected to be owned by a stack, and there doesn't seem to be any inverted dependency resources to add additional backends/urlmaps to an LB. If it's just for CloudRun, you could look into Serverless NEGs which support dynamic registration of services based on their name https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts
You could also make a dynamic provider to add additional path matchers to the urlmap with the gcp sdk for other cases, too. Worth upvoting this issue for support: https://github.com/hashicorp/terraform-provider-google/issues/8662
a

alert-midnight-11504

10/25/2023, 4:26 PM
Hey Anthony thank you for the response. We are utilizing multiple NEGs with 2nd gen cloud functions (uses cloud run under the hood). Not seeing in the docs how / where I could accomplish adding multiple serverless NEGs to a single global external application load balancer https://cloud.google.com/api-gateway/docs/gateway-load-balancing
d

dry-keyboard-94795

10/25/2023, 4:27 PM
It's not well documented, and unfortunately I don't have access to the example I did at work anymore. Supported using URL Masks https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts#url_masks
a

alert-midnight-11504

10/25/2023, 4:29 PM
what u posted is exactly what we are trying to do but but with pulumi. everytime we spin up a new service we need to add the NEG to the backend LB along with URLMap updates so we can route traffic based on headers etc…. IE I see we can accomplish this maybe via gcloud? https://cloud.google.com/load-balancing/docs/https/setting-up-query-and-header-routing#gcloud_6
common Anthony don’t leave me hanging GET that example!!!! 🥺
d

dry-keyboard-94795

10/25/2023, 4:35 PM
Alas, it's scratch code in a git repo I don't have access to anymore; and not local. Going to grab my laptop and do a quick example. Gcp LBs are very... Overloaded 😂
f

freezing-garage-4831

10/25/2023, 4:37 PM
excited
a

alert-midnight-11504

10/25/2023, 4:37 PM
do you take paypal sir?
d

dry-keyboard-94795

10/25/2023, 4:38 PM
all my pulumi support is pro-bono. That way I'm not liable 😉
f

freezing-garage-4831

10/25/2023, 4:43 PM
thinking
a

alert-midnight-11504

10/25/2023, 4:48 PM
I thought about replacing the backend service by fetching the current one, adding the NEG for the service and doing that for every service but am getting a 409 The resource ‘projects/staging/global/backendServices/google-backend-service’ already exists, alreadyExists
Copy code
const backendService = new BackendService('google-backend-service', {
      project,
      loadBalancingScheme: 'EXTERNAL',
      name: 'google-backend-service',
      description: 'Backend service for custom domain API Gateway load balancers',
      logConfig: { enable: true },
      backends: [{
        group: neg.selfLink
      }]
    }, { parent: neg, deleteBeforeReplace: true, replaceOnChanges: ['*'] });
d

dry-keyboard-94795

10/25/2023, 5:01 PM
yep, pulumi expects full ownership of resources
so what you're looking for is:? • ForwardingRule + URLMap in the infra stack • CloudRun service + patch URLMap in other stacks
a

alert-midnight-11504

10/25/2023, 5:10 PM
correct, for example we have two stacks. Google Stack: This is where we put global / foundational infra that can be referenced by other stacks / teams / services. The stack involves the following resources… • BackendService • URLMap • SSLCertificate • TargetHttpsProxy • GlobalForwardingRule Service Stack: This is where we create an API Gateway for the Cloud Run service and add it to the existing backend in the Google Stack. The stack involves he following resources… • Api • Config • Gateway • RegionNetworkEndpointGroup
trying to add the RegionNetworkEndpointGroup created in a child / dependent stack to the parent BackendService owned by the Google infra stack.
d

dry-keyboard-94795

10/25/2023, 5:13 PM
I think you'd create the BackendService for each service, and add it to the Urlmap path rules?
s

stocky-restaurant-98004

10/25/2023, 5:14 PM
This is a real shot in the dark, and I don't have deep expertise in Google Cloud, but I think the resources you mentioned are used in this codebase from a workshop I did with @refined-pilot-45584: https://github.com/timbohiatt/gke-at-scale-pulumi
a

alert-midnight-11504

10/25/2023, 5:14 PM
was hoping just to create one Global Load Balancer for cost purposes
s

stocky-restaurant-98004

10/25/2023, 5:14 PM
Hope that helps.
a

alert-midnight-11504

10/25/2023, 5:15 PM
@dry-keyboard-94795 is it possible to modifying an existing UrlMap? I was running into the same issues as the backend service. I would do that as a compromise.
d

dry-keyboard-94795

10/25/2023, 5:16 PM
@stocky-restaurant-98004 correct; however in this case the services are created in a separate stack to the LB, where the LB is in the parent stack so doesn't have references straight away to the services
a

alert-midnight-11504

10/25/2023, 5:17 PM
Would I run into the same issue if the UrlMap was in the parent stack?
d

dry-keyboard-94795

10/25/2023, 5:17 PM
however pulumi (+ Terraform) don't have this exposed as a resource
a

alert-midnight-11504

10/25/2023, 5:17 PM
@dry-keyboard-94795 that was going to be my fallback but not sure how to run gcloud commands safely during CI/CD dep;loyment with pulumi
d

dry-keyboard-94795

10/25/2023, 5:17 PM
You could do it with a dynamic provider, or 
local.Command
a

alert-midnight-11504

10/25/2023, 5:18 PM
I was trying to do a bit of research on defining a dynamic provider but was having trouble finding a good example close to my use case.
have not seen 
local.command
i can try and look this up real quick
s

stocky-restaurant-98004

10/25/2023, 5:19 PM
Again, vague recollection on my part, but I do believe you have to use 
gcloud
with the Pulumi Command provider to do some of the stuff in Tim's repo because it's not supported in the provider.
d

dry-keyboard-94795

10/25/2023, 5:19 PM
but not sure how to run gcloud commands safely during CI/CD dep;loyment with pulumi
this depends on how you have auth setup. A default setup where you've setup the application-default credentials should work, but gets a little more involved if you're using assumed roles
s

stocky-restaurant-98004

10/25/2023, 5:19 PM
@alert-midnight-11504 If you use TF, 
command.local.Command
in Pulumi often serves a similar purpose to 
null_provider
with a provisioner or whatever it's called.
a

alert-midnight-11504

10/25/2023, 5:20 PM
I think local.command is the way to go.
s

stocky-restaurant-98004

10/25/2023, 5:20 PM
Dropping down to use 
command.local.Command
to use the cloud provider's CLI is not uncommon. I've had to do it with all of the hyperscalers at one point or another.
a

alert-midnight-11504

10/25/2023, 5:21 PM
we are using application default, which is used by Google Cloud Build, and should be able to execute those commands if the service account has access.
d

dry-keyboard-94795

10/25/2023, 5:21 PM
a

alert-midnight-11504

10/25/2023, 5:22 PM
Well… dang. But I think this is the cleanest route to go. Thank you all so much! and the repo you provided @stocky-restaurant-98004 is extremely helpful! Mind if keep yall posted on my progress? Going to try and implement this now.
d

dry-keyboard-94795

10/25/2023, 5:24 PM
sure. I've also just gotten my LB up. sorry it took so long, had to setup a new gcp account 🤣
a

alert-midnight-11504

10/25/2023, 5:24 PM
jeez
ya im a little worried on how little documentation there is around serverless NEGs and API Gateway at Google. Still only supports Swagger 2.0. Having the ability to add a managed gateway a Global External LB just went GA. 😬 🪦
d

dry-keyboard-94795

10/25/2023, 5:27 PM
the lack of docs was one of the reasons why we didn't go ahead with the project. + was in preview at the time
a

alert-midnight-11504

10/25/2023, 5:27 PM
oh great…
f

freezing-garage-4831

10/25/2023, 5:27 PM
trying not to cry
a

alert-midnight-11504

10/25/2023, 5:29 PM
I don’t really have a choice, we have a service that is falling over, and need to split traffic off an already overloaded service / endpoint. The problem is we cannot just take the service down or have our provider hit us on a separate endpoint without the ability to rollback.
d

dry-keyboard-94795

10/25/2023, 5:30 PM
this is why I loved having Cloudflare sat infront of some of our services. Made it really easy to use Workers (ie, real code) to do network migrations
a

alert-midnight-11504

10/25/2023, 5:30 PM
wait!
we are using cloudflare
go on…
d

dry-keyboard-94795

10/25/2023, 5:31 PM
it needs to be in DNS proxy mode. I think you're using GCP Managed certificates, so guessing you're not in proxy mode
a

alert-midnight-11504

10/25/2023, 5:31 PM
we are using self managed ssl certificate from cloudflare 😛
we are in full strict SSL
IE I created an origin ssl cert.
how can i split traffic at the DNS by looking for a particular header?
d

dry-keyboard-94795

10/25/2023, 5:36 PM
if you're on enterprise, you can use an Origin Rule. Otherwise, you need to setup a worker on the hostname. Workers are effectively Express services
a

alert-midnight-11504

10/25/2023, 5:38 PM
not on enterprise 😢. This is too freaking hilarious. The service we have in place today (also using express), tried splitting the traffic by doing a redirect / forward back to Cloudflare on a different route. But we kept getting 504 Gateway timeouts….
so i tried doing the splitting at the LB level, and now it sounds like doing at the DNS with a express service / worker in front of everything is the way to go?
f

freezing-garage-4831

10/25/2023, 5:39 PM
I have no idea what im doing
d

dry-keyboard-94795

10/25/2023, 5:44 PM
it's a way to do it. Worth looking through their examples: https://developers.cloudflare.com/workers/examples/
@alert-midnight-11504 do the services share the same hostname? It seems 
add-path-matcher
requires distinct hostnames. If all services are made in the same stack, this isn't a problem as the path matching rules can be combined
a

alert-midnight-11504

10/25/2023, 6:30 PM
the service urls are different / dynamic when they are stood up / created
the dns subdomain is the same along with the path
I think we are okay on the path matcher meaning, one route goes to 
/webhooks/Authorization
on service A, and the other goes to 
/provider/Authorization
on service B
d

dry-keyboard-94795

10/25/2023, 6:33 PM
and all services share the same domain?
a

alert-midnight-11504

10/25/2023, 6:34 PM
yes same subdomain
d

dry-keyboard-94795

10/25/2023, 6:34 PM
are they all made in the same stack?
a

alert-midnight-11504

10/25/2023, 6:34 PM
to split the traffic one service uses a JWT Bearer Token and the other users a 
signature
header. Thats how we want to split the traffic using route rules
@dry-keyboard-94795 no they are in two seprate stacks. One stack per service infra.
d

dry-keyboard-94795

10/25/2023, 6:47 PM
looks like header matchers can't be added using the CLI without doing the full import, too
a

alert-midnight-11504

10/25/2023, 7:00 PM
meaning I cannot use the GCloud CLI to add urlMaps like in this example? https://cloud.google.com/load-balancing/docs/https/setting-up-url-redirects-classic
d

dry-keyboard-94795

10/25/2023, 7:01 PM
You can update it like that, just increases the complexity of implementation
a

alert-midnight-11504

10/25/2023, 7:02 PM
ill take anything at this point
d

dry-keyboard-94795

10/25/2023, 7:04 PM
if this is a temporary thing while you migrate, I'd be tempted to do it by hand
a

alert-midnight-11504

10/25/2023, 7:05 PM
it is a temporary thing atm, but I was really hoping to have a strategy for bringing on new services from other stacks / teams…
IE we are going to be stuck doing this again once we wrap up another micro service implementation on the same api.company-domain.com
Whats the point of having a global load balancer for 1 service?
d

dry-keyboard-94795

10/25/2023, 7:09 PM
you could reference the service stacks in the infra stack, and update the urlmap during creation
a

alert-midnight-11504

10/25/2023, 7:10 PM
I can’t because the service stacks depend on the google (base) stack. And if Create the LB after all the services are created, then I need to teardown / replace the global loadbalancer anytime a service changes 😬
d

dry-keyboard-94795

10/25/2023, 7:10 PM
the urlmap will update in place
a

alert-midnight-11504

10/25/2023, 7:11 PM
^ what you mean? I might be missing something
d

dry-keyboard-94795

10/25/2023, 7:11 PM
to add a service, you only need to add it in the urlmap's route matcher
a

alert-midnight-11504

10/25/2023, 7:12 PM
are you saying I can update 1 UrlMap reference and create multiple backends?
d

dry-keyboard-94795

10/25/2023, 7:12 PM
yes
the service stacks can create their own Backends + NEGs. so yes, would use multiple backend service resouces
a

alert-midnight-11504

10/25/2023, 7:13 PM
Is this viable at scale? IE multiple Backend Services would mean a LB for each service i guess?
d

dry-keyboard-94795

10/25/2023, 7:13 PM
it'd still be just a single LB (ie, ForwardingRule)
a

alert-midnight-11504

10/25/2023, 7:14 PM
ohhhhhhh
this diagram from google docs is tripping me up. Meaning I cannot find a way to actually implement this diagram
how do i go about updating the urlMap? using local command or is there a pulumi native way of doing this?
d

dry-keyboard-94795

10/25/2023, 7:16 PM
you can use Pulumi. Whenever a new service is added, you can update the code
a

alert-midnight-11504

10/25/2023, 7:17 PM
IE would i 
UrlMap.get()
in the service stack to get a reference from base Google stack and use a 
transformation
?
d

dry-keyboard-94795

10/25/2023, 7:17 PM
do you create the URLMap in pulumi currently?
a

alert-midnight-11504

10/25/2023, 7:18 PM
yes
but in google stack
s

stocky-restaurant-98004

10/25/2023, 7:32 PM
This convo rules. I love watchin' y'all work through a real tough problem!
a

alert-midnight-11504

10/25/2023, 7:37 PM
Hey @stocky-restaurant-98004 mind calling our managing investors and tell them what you just said. Thx! Met you guys at GoogleNext’23 I had nothing but great things to say, and got a free pair of Pulumi Socks ❤️
d

dry-keyboard-94795

10/25/2023, 7:38 PM
heh, is everyone copying M$oft with the socks now? I sitll have my vscode socks
a

alert-midnight-11504

10/25/2023, 7:39 PM
Ya Aiven was handing out 🦀 socks too lol
d

dry-keyboard-94795

10/25/2023, 7:39 PM
so your google stack can contain an urlmap that looks something like this: https://gist.github.com/antdking/9adad9ec7d03d212be9de5f9159bd6cc Note that the backendservice IDs are loaded from the stack references
hastily checks this actually works
a

alert-midnight-11504

10/25/2023, 7:40 PM
this wont work because the service’s stack depend on the Base Google stack which is the code you are presenting.
Copy code
const servicesStack = new pulumi.StackReference(`${org}/gcp-lb-multistack-services/dev`);
// These could be sourced from multiple stacks
const s1BackendServiceId = servicesStack.requireOutput("be1Id");
const s2BackendServiceId = servicesStack.requireOutput("be2Id");
IE the services have not been created yet. they are created once the Google stack has been deployed 😞
d

dry-keyboard-94795

10/25/2023, 7:41 PM
the expectation is you update the code after the service stacks have initially run
a

alert-midnight-11504

10/25/2023, 7:41 PM
I need to go the other way, where I am importing the url map and updating it
But this would create a dependency on the all the services being creating first.
d

dry-keyboard-94795

10/25/2023, 7:42 PM
correct
a

alert-midnight-11504

10/25/2023, 7:42 PM
which means if one of those services changes, gets added, removed, I have to re-create the LB with the new state from the services
😬
d

dry-keyboard-94795

10/25/2023, 7:44 PM
it'll just update the urlmap. If you add this to the 
routeRules
, it'll not recreate everything:
Copy code
{
    service: s2BackendServiceId,
    priority: 3,
    matchRules: [
        {
            prefixMatch: "/s2",
        }
    ]
}
image.png
however as you say, the service stacks must have run before you update the code in your google stack
a

alert-midnight-11504

10/25/2023, 7:47 PM
ya which, im thinking, would be another stack just to manage the LB deployment after all the services have deployed.
d

dry-keyboard-94795

10/25/2023, 7:48 PM
I think that is simpler than trying to update the urlmap inplace from the service stacks at this point
a

alert-midnight-11504

10/25/2023, 7:49 PM
I might try going the urlMap update route first using the local.command CLI then if i get stuck ill try ur approach
d

dry-keyboard-94795

10/25/2023, 7:49 PM
additionally, the service stacks can just export their cloudrun IDs, and have the NEGs + BackendServices created in the google stack too
a

alert-midnight-11504

10/25/2023, 7:49 PM
I am hoping the local command to update the URL map is just a few lines vs implementing and orchestrating another stack deployment life cycle.
d

dry-keyboard-94795

10/25/2023, 7:50 PM
there's a couple of limitations with `add-path-matcher`: • You can only have one pathMatcher per hostname • it doesn't support adding 
routeRules
, which are required for your header-based routing
a

alert-midnight-11504

10/25/2023, 7:52 PM
😬
d

dry-keyboard-94795

10/25/2023, 7:52 PM
to do it with the CLI, you'll need to 
export
the urlmap config, update the yaml with additional path rules, then 
import
it. I'm not sure what protections there are if you say these stacks will run in parallel; though it should be fine as this is only for creates
a

alert-midnight-11504

10/25/2023, 7:53 PM
ill be dead in the water if I cannot route based on headers
@dry-keyboard-94795 I think you approach is the cleanest and makes the most sense. Going to go heads down for a bit and see if i can get this implemented. Ill check-in in a little bit if thats alright. I def owe you a 🍺
Thank you so much for your pro-bono time 😛
d

dry-keyboard-94795

10/25/2023, 8:02 PM
no problem. It's an interesting problem, and I think would be a great feature to be added to the GCP provider in the future. AWS has had support for configuring this kind of routing decentralised since forever: https://www.pulumi.com/registry/packages/aws/api-docs/alb/listenerrule/
a

alert-midnight-11504

10/25/2023, 8:04 PM
ya… last thing I want to do is be like.
d

dry-keyboard-94795

10/25/2023, 8:06 PM
For completeness, I've added the referenced services stack to that gist: https://gist.github.com/antdking/9adad9ec7d03d212be9de5f9159bd6cc
a

alert-midnight-11504

10/25/2023, 8:15 PM
☝️ @dry-keyboard-94795 this is amazing thank you!
s

salmon-account-74572

10/25/2023, 8:40 PM
@alert-midnight-11504 Hey Austin, @stocky-restaurant-98004 mentioned you might be interested in writing something up on this topic. I’d be happy to help support you, collaborate with you, etc., if you decide to go that route. Feel free to DM me.
a

alert-midnight-11504

10/26/2023, 7:34 PM
@dry-keyboard-94795 Looks like your suggestion is working. Just finished implementation, now promoting to staging for additional testing, and then off to production. wish me luck 🙂
Hey @dry-keyboard-94795 @stocky-restaurant-98004 Just wanted to say thank you again. The approach of just tacking on another stack to manage the Global LB and Url Matching rules downstream worked for us. Changes to the services just update the balancer UrlMap instead of just redeploying the whole thing which was critical for us. We have successfully broken up our monolithic webhook service into smaller pieces. We ended up creating a dedicated card authorization service to handle just live card authorizations coming from the card networks. This has resulted in better tracing, logging, monitoring, sizing / scaling configurations. Pulumi Makes it extremely easy to navigate and reason about when making large infrastructure decisions like this. Without IaC and Pulumi’s more developer focused / friendly approach to deploying, testing, and managing changes to our infrastructure over time, this would be much more challenging. Deploying our infrastructure is only half the battle. Building, bundling, deploying, and orchestrating our services from source as part of that IaC process allows our developer’s, who are not necessarily platform / infra experts, to focus on their best work without having to think too much about higher level abstractions at the Cloud Native / GCP Project levels. I really want to do a write-up on this, but we are not quite out of the woods yet. Our initial load tests and new infrastructure as identified other key areas for us to focus our attention as we continue to scale.
Just wanted to give a huge shoutout to @dry-keyboard-94795 and the Pulumi community for being so awesome, approachable, and helpful through these stressful times (for me anyways 😆)
d

dry-keyboard-94795

10/30/2023, 5:30 PM
Thanks, that's great to hear. I'm glad Pulumi has helped you, and hopefully GCP makes urlmaps a little more flexible in the future. We're always here to help
Going with IaC should also help with your compliance requirements. I know our assessors loved it when we said "we just click this button and magic happens" 😂 You'll likely be interested in the latest changes to CrossGuard down the road 
</mercilessUpselling>
🤣
6 Views
Powered by