https://pulumi.com logo
Join Slack
Powered by
I'm trying to install the kube prometheus stack vi...
# kubernetes
w
I'm trying to install the kube prometheus stack via its helm chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack ... I've seen a few people mention it in the slack channels. What transformations are you using to get it to work with pulumi?
Initial approach with no transformations using dotnet/c#:
Copy code
// kube prometheus stack; <https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack>
var kubePrometheusStackCrds = new ConfigGroup("kube-prometheus-stack-crds",
    new ConfigGroupArgs { Yaml = ReadResource("KubePrometheusStackCrds.yaml") },
    new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

var monitoringNs = new Namespace("monitoring",
    new NamespaceArgs { Metadata = new ObjectMetaArgs { Name = "monitoring" } },
    new CustomResourceOptions { Provider = k8sProvider });

var kubePrometheusStackValues =
    new Dictionary<string, object>
    {
        ["alertmanager"] = new
        {
            alertmanagerSpec = new
            {
                nodeSelector = new { role = "monitoring" },
                tolerations = new[] { new { key = "role", @operator = "Exists" } }
            },
            ingress = new { enabled = false }
        },
        ["grafana"] = new
        {
            ingress = new { enabled = false },
            nodeSelector = new { role = "monitoring" },
            tolerations = new[] { new { key = "role", @operator = "Exists" } }
        },
        ["kubeControllerManager"] = new { enabled = false },
        ["kubeEtcd"] = new { enabled = false },
        ["kubeScheduler"] = new { enabled = false },
        ["kube-state-metrics"] = new
        {
            nodeSelector = new { role = "monitoring" },
            tolerations = new[] { new { key = "role", @operator = "Exists" } }
        },
        ["prometheus"] = new
        {
            ingress = new { enabled = false },
            prometheusSpec = new
            {
                podMonitorSelectorNilUsesHelmValues = false,
                serviceMonitorSelectorNilUsesHelmValues = false,
                nodeSelector = new { role = "monitoring" },
                tolerations = new[] { new { key = "role", @operator = "Exists" } }
            }
        },
        ["prometheus-node-exporter"] = new { tolerations = new[] { new { key = "role", @operator = "Exists" } } },
        ["prometheusOperator"] = new
        {
            admissionWebhooks = new
            {
                certManager = new { enabled = true },
                patch = new
                {
                    nodeSelector = new { role = "monitoring" },
                    tolerations = new[] { new { key = "role", @operator = "Exists" } }
                }
            },
            nodeSelector = new { role = "monitoring" },
            tolerations = new[] { new { key = "role", @operator = "Exists" } }
        }
    };

new Chart("kube-prometheus-stack",
    new ChartArgs
    {
        Namespace = monitoringNs.Metadata.Apply(metadata => metadata.Name),
        FetchOptions = new ChartFetchArgs { Repo = "<https://prometheus-community.github.io/helm-charts>" },
        Chart = "kube-prometheus-stack",
        Version = K8sConfig.KubePrometheusStackChartVersion,
        Values = kubePrometheusStackValues,
        SkipCRDRendering = true
    },
    new ComponentResourceOptions { DependsOn = { kubePrometheusStackCrds, certManagerCrds }, Provider = k8sProvider });
Fails with:
Copy code
+  kubernetes:<http://apiextensions.k8s.io/v1:CustomResourceDefinition|apiextensions.k8s.io/v1:CustomResourceDefinition> monitoring/kube-prometheus-stack-kube-state-metrics create error: Duplicate resource URN
'urn:pulumi:alpha::k8s::kubernetes:<http://helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/kube-prometheus-stack-kube-state-metrics';|helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/kube-prometheus-stack-kube-state-metrics';> try giving it a unique name
Full log...
Pulumi-KPS.log
@better-actor-92669 @gorgeous-country-43026?
g
Hi @worried-city-86458. I gave up, couldn't get it working
Ended up integrating into Azure Log Analytics even though it costs a ton
w
b
hey @worried-city-86458.
Copy code
# Do not wait for services in 'kube-system' namespace
# kps-rapid-kube-prometheus-kube-controller-manager
# kps-rapid-kube-prometheus-kube-etcd
# kps-rapid-kube-prometheus-kube-proxy
# kps-rapid-kube-prometheus-kube-scheduler
def kube_system_not_wait(obj):
    if obj["kind"] == "Service":
        if obj["metadata"]["namespace"] == "kube-system":
            if not obj["metadata"].get("annotations"):
                obj["metadata"]["annotations"] =\
                    {
                        "<http://pulumi.com/skipAwait|pulumi.com/skipAwait>": "true"
                    }


# Apply 'DaemonSet(s)' only to 'linux' nodes
def only_linux_nodes(obj):
    if obj["kind"] == "DaemonSet":
        obj["spec"]["template"]["spec"]["nodeSelector"] =\
            {
                "<http://beta.kubernetes.io/os|beta.kubernetes.io/os>": "linux"
            }


# TODO: Remove this when <https://github.com/pulumi/pulumi-kubernetes/issues/800>
#  is fixed
# Remove the .status field from CRDs
def remove_crds_status(obj):
    if obj["kind"] == "CustomResourceDefinition":
        del obj["status"]


# Do not install CRDs as they are installed via ConfigFile
def omit_crds(obj):
    if obj["kind"] == "CustomResourceDefinition":
        obj["apiVersion"] = "v1"
        obj["kind"] = "List"


# TODO: Remove this after it is fixed
#  <https://github.com/pulumi/pulumi-kubernetes/issues/555>
# Remove Grafana's 'test-pod'
def omit_grafana_test_pod(obj):
    # if obj["kind"] == "Pod" and 'grafana-test' in obj["metadata"]["name"]:
    if 'grafana-test' in obj["metadata"]["name"]:
        obj["apiVersion"] = "v1"
        obj["kind"] = "List"
w
Thanks, @better-actor-92669. I got it to work with a few differences to the above. In particular I didn't need the 
skipAwait
annotation. Why did you need that?
Copy code
new Chart("kube-prometheus-stack",
    new ChartArgs
    {
        Namespace = monitoringNs.Metadata.Apply(metadata => metadata.Name),
        FetchOptions = new ChartFetchArgs { Repo = "<https://prometheus-community.github.io/helm-charts>" },
        Chart = "kube-prometheus-stack",
        Version = K8sConfig.KubePrometheusStackChartVersion,
        Values = kubePrometheusStackValues,
        SkipCRDRendering = true,
        Transformations =
        {
            (obj, _) => obj.MutateWhenKind("PodSecurityPolicy", mutable => mutable.OmitNamespace()),
            (obj, _) => obj.MutateWhenKind("AlertManager", mutable => mutable.SetName(name => $"{name}-am")),
            (obj, _) => obj.MutateWhenKind("PrometheusRule", mutable => mutable.SetName(name => $"{name}-pr")),
            (obj, _) => obj.MutateWhenKind("ServiceMonitor", mutable => mutable.SetName(name => $"{name}-sm")),
            (obj, _) => obj.MutateWhenName("kps-grafana-test", mutable => mutable.Exclude())
        }
    },
    new ComponentResourceOptions { DependsOn = { kubePrometheusStackCrds, certManagerCrds }, Provider = k8sProvider });
The main issue really is the non-unique names for crd instances - I had to tweak the names to force pulumi to generate unique urns.
@gorgeous-egg-16927 this is a long standing issue that really needs to be fixed. I'm confused as to why pulumi is not using the crd kind instead of the generic 
CustomResourceDefinition
as that would solve it.
b
@worried-city-86458, I don't remember to be honest, but my best guess is that some of the Services in kube-system namespace wouldn't be marked as "ready" by Pulumi ("pulumi up" was timing out), so I had to bypass that manually
6 Views
Open in Slack
PreviousNext
Powered by