https://pulumi.com logo
e

early-intern-90238

10/03/2019, 11:08 PM
Hey guys, quick opinionated question. If your using Kubernetes do you prefer a seperate dev environment, I am currently managing dev with labels but I've had some scenarious where this has been pretty problematic. So I am thinking of using an entirely seperate cluster for dev, stage, and only share canary with production. As for developer machines, is anyone using tilt?
c

cool-egg-852

10/03/2019, 11:42 PM
We use 3 environments: development, staging, production. None of them are accessible by our engineering staff, only cloud operations (devops + infra). Each environment is a separate google project, and we have 1 cluster each. This ensures that nothing in one environment can touch another.
I do highly recommend the separate google project, if possible, but definitely the separate cluster. A bad deploy of something like istio can bring down all of your applications. Having a way to test it out first is really handy.
e

early-intern-90238

10/04/2019, 12:34 AM
Yeah, this is how I've traditionally do it. I know alot of recent advice says you can just use one but Im not buying it. I'm going to agree with you and use seperate projects. Makes more sense to me...
Also I think I can setup dev and stage to just scale down to 0 using Preemptive nodes if I want to really optimize cost.
b

better-rainbow-14549

10/04/2019, 9:55 AM
we also do entirely separate - different azure subscriptions and clusters. as nesl247 says it's easy to break a whole kube cluster and anything with CRDs or a daemonset is going to cause interference
c

cool-egg-852

10/04/2019, 2:00 PM
Basically I see it as this: You either pay to maintain 3 separate environments, or you pay when your production environment gets broken. One is a known/fixed cost, the other, who knows.
Either way, you pay.
Also, if you don’t have separate environments,you can’t test k8s upgrades themselves.
c

clean-engineer-75963

10/04/2019, 4:51 PM
We keep dev and stage together, but separate from production, at the cluster level. We keep dev and stage separate with namespaces, rather than labels. I think you'll get much stronger separation if you separate the environments that way.
We do eventually plan to separate staging from dev into its own cluster.
b

better-rainbow-14549

10/04/2019, 4:54 PM
one of the benefits of pulumi is it made scaling down one cluster and bringing up two separate ones instead essentially effortless - for the same azure VM cost
we just have a one node cluster for qa/staging
e

early-intern-90238

10/06/2019, 8:06 PM
that's also true, you could tear down and spin up dev and qa as needed.