Hey guys, quick opinionated question. If your using Kubernetes do you prefer a seperate dev environment, I am currently managing dev with labels but I've had some scenarious where this has been pretty problematic. So I am thinking of using an entirely seperate cluster for dev, stage, and only share canary with production. As for developer machines, is anyone using tilt?

We use 3 environments: development, staging, production. None of them are accessible by our engineering staff, only cloud operations (devops + infra). Each environment is a separate google project, and we have 1 cluster each. This ensures that nothing in one environment can touch another.

Yeah, this is how I've traditionally do it. I know alot of recent advice says you can just use one but Im not buying it. I'm going to agree with you and use seperate projects. Makes more sense to me...

we also do entirely separate - different azure subscriptions and clusters. as nesl247 says it's easy to break a whole kube cluster and anything with CRDs or a daemonset is going to cause interference

Basically I see it as this: You either pay to maintain 3 separate environments, or you pay when your production environment gets broken. One is a known/fixed cost, the other, who knows.

We keep dev and stage together, but separate from production, at the cluster level. We keep dev and stage separate with namespaces, rather than labels. I think you'll get much stronger separation if you separate the environments that way.

one of the benefits of pulumi is it made scaling down one cluster and bringing up two separate ones instead essentially effortless - for the same azure VM cost

that's also true, you could tear down and spin up dev and qa as needed.