https://pulumi.com logo
#general
Title
# general
h

hundreds-portugal-17080

12/24/2019, 7:27 PM
Hi, We are using statefulset(https://github.com/elastic/helm-charts/blob/7.5.0/elasticsearch/templates/statefulset.yaml) and spinning up eks through Pulumi code. It is creating aws ebs volumes and everything is working fine. Scenario is that, When the perform pulumi destroy (bringing down the cluster) and bring it up again(pulumi up ), statefulset is creating new volumes instead of reusing existing ones. We are in testing phase where we don't want to keep 5 node cluster running all the time and at the same time, we don't want to recreate the data all the time. So, we would like to shutdown and start when ever needed. Is there is any approach you could recommend to re-use existing ebs volumes. Also, in real scenarios, Eks can go down for n number of reasons and it is very important for us to address reusing volumes. Please advise.
c

creamy-potato-29402

12/24/2019, 7:33 PM
that’s just how kubernetes works.
it’s not specific to pulumi.
if you want to re-use some random volume from somewhere, you have to configure the statefulset to use it.
h

hundreds-portugal-17080

12/24/2019, 7:38 PM
thanks for quick response. we want it to be automated way. I am reading a bit about etcd.. is it possible ? https://stackoverflow.com/questions/44929600/kubernetes-statefulsets-after-cluster-restart
c

creamy-potato-29402

12/24/2019, 7:40 PM
b

broad-dog-22463

12/24/2019, 7:40 PM
Is this worth something in examples and a blog post @creamy-potato-29402 ? :)
c

creamy-potato-29402

12/24/2019, 7:41 PM
possibly, but I (and, I think, others in the community) consider statefulset to be kind of a corner case. so the interest might be limited.
generally the guidance is: use sts if you really can’t do something else.
b

broad-dog-22463

12/24/2019, 7:44 PM
Gotcha
c

creamy-potato-29402

12/24/2019, 7:44 PM
a blog post about what those gotchas are, though, is probably of interest.
like, why that’s true.
also what the heck are you doing online??
h

hundreds-portugal-17080

12/24/2019, 7:45 PM
what is your alternative for statefulset then.
c

creamy-potato-29402

12/24/2019, 7:46 PM
depends what you’re doing. are you running a database, writing to the FS, something else?
h

hundreds-portugal-17080

12/24/2019, 7:47 PM
I am running elasticsearch writing indexed data that needs to be persisted which is used by kibana and grafna to read
c

creamy-potato-29402

12/24/2019, 7:54 PM
not an ES expert, but it depends how the block store is implemented. one big problem with block stores generally is that they do not understand the replication semantics of whatever is built on top. for example, if ES has a replication log of some sort, and ES disappears, the block store doesn’t understand the replication log, and can’t recover it after the fact for you.
generally speaking, most databases expect to replicate writes on their own. if the underlying block store does that on your behalf, that is a different contract with a different set of failure modes. the block store doesn’t know who is elected leader, and so on.
and so you can get into tricky situations where your pod goes away and then the rep. log is being replicated, but block store drops in-flight writes and you lose data as a result.
in general, for databases, my recommendation is: • Use managed services. • If you can’t, use VMs with local disks. • If you can’t, write your service to correctly use the kube primitives. • If you can’t, use statefulsets and budget for data loss.
👍 1
h

hundreds-portugal-17080

12/24/2019, 8:06 PM
ok. Thank you very much
s

steep-toddler-94095

12/24/2019, 10:57 PM
why not scale down the EKS cluster to 0 worker nodes when not in use, and scale back up when you need it
h

hundreds-portugal-17080

12/25/2019, 9:15 PM
awesome option.. let me give it a try
it works. thanks.
f

flat-insurance-25294

12/26/2019, 3:20 PM
@lemon-student-18495 To be fair, you’re stilling paying $140 a month for the control plane.
I also second @creamy-potato-29402 I am deadly afraid of storing our own persistent data, hell, scared to even host Redis ourselves.
s

steep-toddler-94095

12/26/2019, 6:19 PM
the $140/month may be worth it, depending on his budget and configuration. It's a hassle to deal with configuring volume attachment manually, especially if he wants to scale the ES cluster in future tests, or add zone-awareness.