This may be a little of topic, but I think the community here must have some insight on this.
I just updated a pulumi stack which deploys the https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack helm chart
If you have a look at that page you'll see that major version upgrades require the update of the CRD as helm doesn't support updating CRD, they provide yaml to do it.
I found out about this the hard way by having my helm chart deploy fail, which amongst other things deleted the PV and PVC that grafana use and have all the dashboards I wrote on them. :'(
So what do people do about such upgrades? Do you pin all the helm chart versions and then work out a release process for each major version upgrade? Do you backup all your PVs etc. before every upgrade, 'just in case'.
I'm trying to gain confidence in the Pulumi/k8s/helm ecosystem but it seems to be full of footguns. 😕
My route forward is to create the PV and PVC before the helm release and then use a feature of this chart which allows me to specify an existing PVC for use for grafana's persistence. I can then set
on it and it should be safe.
However this doesn't resolve the bigger issue with CRDs not being updated by helm and major version upgrades of helm charts which use them requiring special attention to run CRD updates which seem hard to automate, and destined to break your deployed monitoring stack if you're not careful.