I started following the same steps I used to set up the AKS service principal which has too much rights (provisioning load balancers and network resources).
I think that this is what I actually need:
https://www.pulumi.com/docs/guides/crosswalk/kubernetes/configure-access-control/
Just need to adapt it to our env, test it a bit, figure out how to get the kubeconfig and its probably gonna be half past midnight again 🤷♂️