I've done some investigation on how to create a CD-setup, and I think that I've found a good solution using multiple stacks for identity/infrastructure/app-deployments for a monorepo-setup. We are however now trying to split up some of our code into smaller repositories, and that is when I think it will become somewhat tricky to handle. My idea is to have a separate repo for the pulumi infrastructure configuration, where I can create database instances, k8s-cluster (with external IP, ingress, ssl-cert) and more. In the repo for each service (k8s-deployment) I would like to get hold of the Ingress-definition from the shared infra-stack, and modify the ingress-rule-spec according to each service's needs. Would this be possible, or should I try to modify the structure in any other way? (I'm using GCP/GKE and Typescript)

I’d recommend checking out https://www.pulumi.com/docs/guides/crosswalk/kubernetes/ It has detailed examples across all the major clouds, including example projects and stacks on the linked GitHub repos.

I've taken a look at the examples in the playbook, and it matches most of my current PoC. What I'm missing is how the Ingress rules are defined? I found the ingress-namespace and ingress-roles, but not the actual Ingress definition? According to the schematic image, shouldn't the Ingress be defined in the app-services-layer? I've probably misunderstood some details? https://github.com/pulumi/kubernetes-guides/blob/master/gcp/05-app-services/index.ts

cc @breezy-hamburger-69619

A more complete example with Ingress objects can be found at [1], which puts nginx into the app services stack. This step was general to any k8s cluster so we did not put it in the provider specific app services, sorry about the confusion! 1 - https://github.com/pulumi/kubernetes-guides/blob/master/general-app-services/nginx-ingress-controller/index.ts

Thanks for the pointer! I'm still wondering if it would be possible to share one ingress-instance, defined in the app-services-stack, and let each deployment in the app-layer add their own rules? My aim is that each repo/build-definition should handle all parts of their deployment? If I define the ingress in app-services-stack, I need to know in advance which deployments that will be deployed in the same cluster? Since the names of the deployments are unique for every new instance, that could be tricky to figure out?

The linked example should not have the deployment+svc+ingress in the app services, only nginx. They were co-located in the same stack in the off-chance the nginx ingress controller was not deployed in the walk through. But yes, you can share an nginx ingress controller instance. The default ingress class in the chart requires ingress objects be annotated with

{"<http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>": "nginx"}

for this purpose. This can be reconfigured if need be in the nginx instance. You would deploy nginx at this layer and then for your apps keep them in an apps stack co-located with their rules and ingress objects annotated with the class to use, which will be picked up by the nginx ing cntlr configured to watch for that class. https://github.com/pulumi/kubernetes-guides/blob/master/general-app-services/nginx-ingress-controller/index.ts#L79

I'm currently using the GCP ingress, could that also work, or do I need to install the nginx-controller? Do you mean that there will be several Ingress-instances, one for each app? As far as I understand, I can only use one global-static-ip by one Ingress/L7 Loadbalancer?

If you’re using the GCP ingress controller you do not need to install the nginx-ingress-controller.

Do you mean that there will be several Ingress-instances, one for each app?

I meant that a single nginx-ingress-controller can serve many Ingress objects as long as the class annotation matches what the controller is configured to use.

As far as I understand, I can only use one global-static-ip by one Ingress/L7 Loadbalancer?

Yes, as I understand only a single globoal static ip can be associated with a single ingress object [1][2]. By default ingress objects backed by the GCP L7 ingress controller use ephemeral IPs. It may serve you to use this default setting, and use something like

externalDNS

which will automatically update DNS with the ingress IP 1 - https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#step_5_optional_configure_a_static_ip_address 2 - https://cloud.google.com/kubernetes-engine/docs/tutorials/configuring-domain-name-static-ip#step_2b_using_an_ingress

Thanks for your clarifications, but I'm still not getting any closer to enlightenment regarding my initial question! 🙂 Would it be possible to create a setup with a shared Ingress in app-services-stack, and let multiple app-stacks modify the ingress-rules? Another solution that one could think of is to define all Ingress-rules in the app-services-stack, but the nest question would then be how I should get the names of all k8s-services for each app?

Sorry to hear you have not been able to resolve this.

Would it be possible to create a setup with a shared Ingress in app-services-stack, and let multiple app-stacks modify the ingress-rules?

Are you still wanting to use the nginx-ing-controller or the GCP L7 ingress controller?

I guess that the GCP-controller would be simpler to setup? We are now using the GCP controller, which lets us use a ManagedCertificate together with static ip-address, but that is just my initial setup. I guess there are a lot of other alternatives! I would like to aim for a setup using as much as possible of the available GCP resources.

Yes, there are lots of options for ingress management. For GCP, the ingress docs are the best spot to learn how to work with the GKE Ingress controller: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress

Would it be possible to create a setup with a shared Ingress in app-services-stack, and let multiple app-stacks modify the ingress-rules?

Another solution that one could think of is to define all Ingress-rules in the app-services-stack, but the nest question would then be how I should get the names of all k8s-services for each app?

Don’t worry about the app-services-stack if you’re using the GKE ingress controller. Simply model the ingress co-living with the deployment as shown in [1], but put it in your apps stack, not the app-services. Ingress objects reference the service of a deployment so it’s common to co-manage them together 1 - https://github.com/pulumi/kubernetes-guides/blob/master/general-app-services/nginx-ingress-controller/index.ts#L31-L101