https://pulumi.com logo
Powered by
Title
w

worried-city-86458

07/01/2021, 11:31 PM
I'm trying to install the kube prometheus stack via its helm chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack ... I've seen a few people mention it in the slack channels. What transformations are you using to get it to work with pulumi?
Initial approach with no transformations using dotnet/c#: 
// kube prometheus stack; <https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack>
var kubePrometheusStackCrds = new ConfigGroup("kube-prometheus-stack-crds",
    new ConfigGroupArgs { Yaml = ReadResource("KubePrometheusStackCrds.yaml") },
    new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

var monitoringNs = new Namespace("monitoring",
    new NamespaceArgs { Metadata = new ObjectMetaArgs { Name = "monitoring" } },
    new CustomResourceOptions { Provider = k8sProvider });

var kubePrometheusStackValues =
    new Dictionary<string, object>
    {
        ["alertmanager"] = new
        {
            alertmanagerSpec = new
            {
                nodeSelector = new { role = "monitoring" },
                tolerations = new[] { new { key = "role", @operator = "Exists" } }
            },
            ingress = new { enabled = false }
        },
        ["grafana"] = new
        {
            ingress = new { enabled = false },
            nodeSelector = new { role = "monitoring" },
            tolerations = new[] { new { key = "role", @operator = "Exists" } }
        },
        ["kubeControllerManager"] = new { enabled = false },
        ["kubeEtcd"] = new { enabled = false },
        ["kubeScheduler"] = new { enabled = false },
        ["kube-state-metrics"] = new
        {
            nodeSelector = new { role = "monitoring" },
            tolerations = new[] { new { key = "role", @operator = "Exists" } }
        },
        ["prometheus"] = new
        {
            ingress = new { enabled = false },
            prometheusSpec = new
            {
                podMonitorSelectorNilUsesHelmValues = false,
                serviceMonitorSelectorNilUsesHelmValues = false,
                nodeSelector = new { role = "monitoring" },
                tolerations = new[] { new { key = "role", @operator = "Exists" } }
            }
        },
        ["prometheus-node-exporter"] = new { tolerations = new[] { new { key = "role", @operator = "Exists" } } },
        ["prometheusOperator"] = new
        {
            admissionWebhooks = new
            {
                certManager = new { enabled = true },
                patch = new
                {
                    nodeSelector = new { role = "monitoring" },
                    tolerations = new[] { new { key = "role", @operator = "Exists" } }
                }
            },
            nodeSelector = new { role = "monitoring" },
            tolerations = new[] { new { key = "role", @operator = "Exists" } }
        }
    };

new Chart("kube-prometheus-stack",
    new ChartArgs
    {
        Namespace = monitoringNs.Metadata.Apply(metadata => metadata.Name),
        FetchOptions = new ChartFetchArgs { Repo = "<https://prometheus-community.github.io/helm-charts>" },
        Chart = "kube-prometheus-stack",
        Version = K8sConfig.KubePrometheusStackChartVersion,
        Values = kubePrometheusStackValues,
        SkipCRDRendering = true
    },
    new ComponentResourceOptions { DependsOn = { kubePrometheusStackCrds, certManagerCrds }, Provider = k8sProvider });
Fails with: 
+  kubernetes:<http://apiextensions.k8s.io/v1:CustomResourceDefinition|apiextensions.k8s.io/v1:CustomResourceDefinition> monitoring/kube-prometheus-stack-kube-state-metrics create error: Duplicate resource URN
'urn:pulumi:alpha::k8s::kubernetes:<http://helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/kube-prometheus-stack-kube-state-metrics';|helm.sh/v3:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/kube-prometheus-stack-kube-state-metrics';> try giving it a unique name
Full log...
Pulumi-KPS.log
@better-actor-92669 @gorgeous-country-43026?
g

gorgeous-country-43026

07/02/2021, 5:15 AM
Hi @worried-city-86458. I gave up, couldn't get it working
Ended up integrating into Azure Log Analytics even though it costs a ton
w

worried-city-86458

07/02/2021, 6:45 PM
https://github.com/pulumi/pulumi-kubernetes/issues/1225#issuecomment-873182095
b

better-actor-92669

07/05/2021, 8:19 AM
hey @worried-city-86458. 
# Do not wait for services in 'kube-system' namespace
# kps-rapid-kube-prometheus-kube-controller-manager
# kps-rapid-kube-prometheus-kube-etcd
# kps-rapid-kube-prometheus-kube-proxy
# kps-rapid-kube-prometheus-kube-scheduler
def kube_system_not_wait(obj):
    if obj["kind"] == "Service":
        if obj["metadata"]["namespace"] == "kube-system":
            if not obj["metadata"].get("annotations"):
                obj["metadata"]["annotations"] =\
                    {
                        "<http://pulumi.com/skipAwait|pulumi.com/skipAwait>": "true"
                    }


# Apply 'DaemonSet(s)' only to 'linux' nodes
def only_linux_nodes(obj):
    if obj["kind"] == "DaemonSet":
        obj["spec"]["template"]["spec"]["nodeSelector"] =\
            {
                "<http://beta.kubernetes.io/os|beta.kubernetes.io/os>": "linux"
            }


# TODO: Remove this when <https://github.com/pulumi/pulumi-kubernetes/issues/800>
#  is fixed
# Remove the .status field from CRDs
def remove_crds_status(obj):
    if obj["kind"] == "CustomResourceDefinition":
        del obj["status"]


# Do not install CRDs as they are installed via ConfigFile
def omit_crds(obj):
    if obj["kind"] == "CustomResourceDefinition":
        obj["apiVersion"] = "v1"
        obj["kind"] = "List"


# TODO: Remove this after it is fixed
#  <https://github.com/pulumi/pulumi-kubernetes/issues/555>
# Remove Grafana's 'test-pod'
def omit_grafana_test_pod(obj):
    # if obj["kind"] == "Pod" and 'grafana-test' in obj["metadata"]["name"]:
    if 'grafana-test' in obj["metadata"]["name"]:
        obj["apiVersion"] = "v1"
        obj["kind"] = "List"
w

worried-city-86458

07/06/2021, 2:15 AM
Thanks, @better-actor-92669. I got it to work with a few differences to the above. In particular I didn't need the 
skipAwait
annotation. Why did you need that?
new Chart("kube-prometheus-stack",
    new ChartArgs
    {
        Namespace = monitoringNs.Metadata.Apply(metadata => metadata.Name),
        FetchOptions = new ChartFetchArgs { Repo = "<https://prometheus-community.github.io/helm-charts>" },
        Chart = "kube-prometheus-stack",
        Version = K8sConfig.KubePrometheusStackChartVersion,
        Values = kubePrometheusStackValues,
        SkipCRDRendering = true,
        Transformations =
        {
            (obj, _) => obj.MutateWhenKind("PodSecurityPolicy", mutable => mutable.OmitNamespace()),
            (obj, _) => obj.MutateWhenKind("AlertManager", mutable => mutable.SetName(name => $"{name}-am")),
            (obj, _) => obj.MutateWhenKind("PrometheusRule", mutable => mutable.SetName(name => $"{name}-pr")),
            (obj, _) => obj.MutateWhenKind("ServiceMonitor", mutable => mutable.SetName(name => $"{name}-sm")),
            (obj, _) => obj.MutateWhenName("kps-grafana-test", mutable => mutable.Exclude())
        }
    },
    new ComponentResourceOptions { DependsOn = { kubePrometheusStackCrds, certManagerCrds }, Provider = k8sProvider });
The main issue really is the non-unique names for crd instances - I had to tweak the names to force pulumi to generate unique urns.
@gorgeous-egg-16927 this is a long standing issue that really needs to be fixed. I'm confused as to why pulumi is not using the crd kind instead of the generic 
CustomResourceDefinition
as that would solve it.
b

better-actor-92669

07/06/2021, 8:13 AM
@worried-city-86458, I don't remember to be honest, but my best guess is that some of the Services in kube-system namespace wouldn't be marked as "ready" by Pulumi ("pulumi up" was timing out), so I had to bypass that manually
4 Views
#kubernetesJoin Slack
Powered by