Hi, We are using statefulset(https://github.com/elastic/helm-charts/blob/7.5.0/elasticsearch/templates/statefulset.yaml) and spinning up eks through Pulumi code. It is creating aws ebs volumes and everything is working fine. Scenario is that, When the perform pulumi destroy (bringing down the cluster) and bring it up again(pulumi up ), statefulset is creating new volumes instead of reusing existing ones. We are in testing phase where we don't want to keep 5 node cluster running all the time and at the same time, we don't want to recreate the data all the time. So, we would like to shutdown and start when ever needed. Is there is any approach you could recommend to re-use existing ebs volumes. Also, in real scenarios, Eks can go down for n number of reasons and it is very important for us to address reusing volumes. Please advise.

that’s just how kubernetes works.

thanks for quick response. we want it to be automated way. I am reading a bit about etcd.. is it possible ? https://stackoverflow.com/questions/44929600/kubernetes-statefulsets-after-cluster-restart

no, that answer is way off track. try this: https://stackoverflow.com/questions/49729461/add-persistent-volume-in-kubernetes-statefulset

Is this worth something in examples and a blog post @creamy-potato-29402 ? :)

possibly, but I (and, I think, others in the community) consider statefulset to be kind of a corner case. so the interest might be limited.

Gotcha

a blog post about what those gotchas are, though, is probably of interest.

what is your alternative for statefulset then.

depends what you’re doing. are you running a database, writing to the FS, something else?

I am running elasticsearch writing indexed data that needs to be persisted which is used by kibana and grafna to read

not an ES expert, but it depends how the block store is implemented. one big problem with block stores generally is that they do not understand the replication semantics of whatever is built on top. for example, if ES has a replication log of some sort, and ES disappears, the block store doesn’t understand the replication log, and can’t recover it after the fact for you.

ok. Thank you very much

why not scale down the EKS cluster to 0 worker nodes when not in use, and scale back up when you need it

awesome option.. let me give it a try

@lemon-student-18495 To be fair, you’re stilling paying $140 a month for the control plane.

the $140/month may be worth it, depending on his budget and configuration. It's a hassle to deal with configuring volume attachment manually, especially if he wants to scale the ES cluster in future tests, or add zone-awareness.