https://pulumi.com logo
Join Slack
Powered by
Hi, We are using statefulset(<https://github.com/e...
# general
h
Hi, We are using statefulset(https://github.com/elastic/helm-charts/blob/7.5.0/elasticsearch/templates/statefulset.yaml) and spinning up eks through Pulumi code. It is creating aws ebs volumes and everything is working fine. Scenario is that, When the perform pulumi destroy (bringing down the cluster) and bring it up again(pulumi up ), statefulset is creating new volumes instead of reusing existing ones. We are in testing phase where we don't want to keep 5 node cluster running all the time and at the same time, we don't want to recreate the data all the time. So, we would like to shutdown and start when ever needed. Is there is any approach you could recommend to re-use existing ebs volumes. Also, in real scenarios, Eks can go down for n number of reasons and it is very important for us to address reusing volumes. Please advise.
c
that’s just how kubernetes works.
it’s not specific to pulumi.
if you want to re-use some random volume from somewhere, you have to configure the statefulset to use it.
h
thanks for quick response. we want it to be automated way. I am reading a bit about etcd.. is it possible ? https://stackoverflow.com/questions/44929600/kubernetes-statefulsets-after-cluster-restart
c
b
Is this worth something in examples and a blog post @creamy-potato-29402 ? :)
c
possibly, but I (and, I think, others in the community) consider statefulset to be kind of a corner case. so the interest might be limited.
generally the guidance is: use sts if you really can’t do something else.
b
Gotcha
c
a blog post about what those gotchas are, though, is probably of interest.
like, why that’s true.
also what the heck are you doing online??
h
what is your alternative for statefulset then.
c
depends what you’re doing. are you running a database, writing to the FS, something else?
h
I am running elasticsearch writing indexed data that needs to be persisted which is used by kibana and grafna to read
c
not an ES expert, but it depends how the block store is implemented. one big problem with block stores generally is that they do not understand the replication semantics of whatever is built on top. for example, if ES has a replication log of some sort, and ES disappears, the block store doesn’t understand the replication log, and can’t recover it after the fact for you.
generally speaking, most databases expect to replicate writes on their own. if the underlying block store does that on your behalf, that is a different contract with a different set of failure modes. the block store doesn’t know who is elected leader, and so on.
and so you can get into tricky situations where your pod goes away and then the rep. log is being replicated, but block store drops in-flight writes and you lose data as a result.
in general, for databases, my recommendation is: • Use managed services. • If you can’t, use VMs with local disks. • If you can’t, write your service to correctly use the kube primitives. • If you can’t, use statefulsets and budget for data loss.
👍 1
h
ok. Thank you very much
s
why not scale down the EKS cluster to 0 worker nodes when not in use, and scale back up when you need it
h
awesome option.. let me give it a try
it works. thanks.
f
@lemon-student-18495 To be fair, you’re stilling paying $140 a month for the control plane.
I also second @creamy-potato-29402 I am deadly afraid of storing our own persistent data, hell, scared to even host Redis ourselves.
s
the $140/month may be worth it, depending on his budget and configuration. It's a hassle to deal with configuring volume attachment manually, especially if he wants to scale the ES cluster in future tests, or add zone-awareness.
Open in Slack
PreviousNext
Powered by