https://pulumi.com logo
Join Slack
Powered by
FYI - <https://pulumi-community.slack.com/archives...
# kubernetes
s
w
I'll have a play with this soon. Does this mean pulumi k8s will call 
helm uninstall
for the chart resource delete operation? (That would help dependency ordering issues where a chart creates resources outside of k8s, like in aws via a controller, and the rest of the non-k8s stack can't be torn down unless these resources are also deleted.)
s
To be clear, the chart resource is not encapsulated by the helm release resource and nothing changes there. If you use the helm release resource, the 
create
will do the equivalent of 
helm install
(though this directly calls into Helm as a library) and delete will do an equivalent of 
helm uninstall
. This resource explicitly models a 
release
while chart is a component resource that doesn't materialize a release in any way on the cluster. The blog post goes into some detail
👍 1
w
First attempt to move everything from helm chart to helm release was easy at the source level but it blows up at runtime
For example, installing cert manager before using dotnet/c#:
Copy code
// cert manager; <https://github.com/jetstack/cert-manager/tree/master/deploy/charts/cert-manager>
Logger.LogDebug("Installing cert manager");
var certManagerCrds = new ConfigGroup("cert-manager-crds",
    new ConfigGroupArgs { Yaml = ReadResource("CertManagerCrds.yaml") },
    new ComponentResourceOptions { /*Protect = true,*/ Provider = k8sProvider });

var certManagerNs = new Namespace("cert-manager",
    new NamespaceArgs { Metadata = new ObjectMetaArgs { Name = "cert-manager" } },
    new CustomResourceOptions { Provider = k8sProvider });

var certManagerValues =
    new Dictionary<string, object>
    {
        ["prometheus"] = new
        {
            enabled = true,
            servicemonitor = new { enabled = true }
        }
    };

var certManagerChart = new Chart("cert-manager",
    new ChartArgs
    {
        Namespace = "cert-manager",
        FetchOptions = new ChartFetchArgs { Repo = "<https://charts.jetstack.io>" },
        Chart = "cert-manager",
        Version = K8sConfig.CertManagerChartVersion,
        Values = certManagerValues,
        SkipCRDRendering = true
    },
    new ComponentResourceOptions { DependsOn = { certManagerCrds, certManagerNs, kubePrometheusStackCrds }, Provider = k8sProvider });
After (showing just diff from chart to release):
Copy code
var certManagerRelease = new Release("cert-manager",
    new ReleaseArgs
    {
        Namespace = "cert-manager",
        RepositoryOpts = new RepositoryOptsArgs { Repo = "<https://charts.jetstack.io>" },
        Chart = "cert-manager",
        Version = K8sConfig.CertManagerChartVersion,
        Values = certManagerValues,
        SkipCrds = true
    },
    new CustomResourceOptions { DependsOn = { certManagerCrds, certManagerNs, kubePrometheusStackCrds }, Provider = k8sProvider });
However, helm release values is not compatible with helm chart values as it doesn't like my use of anonymous types in the values:
Copy code
Diagnostics:
 
pharos/k8s/alpha (pulumi:pulumi:Stack)
...
error: Running program 'D:\Devel\Mps\devops-gemini-pulumi\Cli\bin\Debug\gemini.dll' failed with an unhandled exception:
System.InvalidOperationException: <>f__AnonymousType15`2[[System.Boolean, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[<>f__AnonymousType7`1[[System.Boolean, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], K8s, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]] is not a supported argument type.
	resource:cert-manager[kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release>].values.id.prometheus
   at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources) in /_/sdk/dotnet/Pulumi/Serialization/Serializer.cs:line 242
   at async Task<ImmutableDictionary<string, object>> Pulumi.Serialization.Serializer.SerializeDictionaryAsync(string ctx, IDictionary dictionary, bool keepResources) in /_/sdk/dotnet/Pulumi/Serialization/Serializer.cs:line 362
   at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources) in /_/sdk/dotnet/Pulumi/Serialization/Serializer.cs:line 221
   at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources) in /_/sdk/dotnet/Pulumi/Serialization/Serializer.cs:line 150
   at async Task<object> Pulumi.Serialization.Serializer.SerializeAsync(string ctx, object prop, bool keepResources) in /_/sdk/dotnet/Pulumi/Serialization/Serializer.cs:line 108
   at async Task<SerializationResult> Pulumi.Deployment.SerializeFilteredPropertiesAsync(string label, IDictionary<string, object> args, Predicate<string> acceptKey, bool keepResources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Serialization.cs:line 53
   at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 32
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
   at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
   at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 153
   at async Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedResourceUrnsAsync(HashSet<Resource> resources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 163
   at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 80
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
   at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
   at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 153
   at async Task<HashSet<string>> Pulumi.Deployment.GetAllTransitivelyReferencedResourceUrnsAsync(HashSet<Resource> resources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 163
   at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 80
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
   at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
   at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 153
   at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
   at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 153
   at async Task<PrepareResult> Pulumi.Deployment.PrepareResourceAsync(string label, Resource res, bool custom, bool remote, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_Prepare.cs:line 42
   at async Task<OutputData<object>> Pulumi.Output<T>.Pulumi.IOutput.GetDataAsync() in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 164
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.RegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_RegisterResource.cs:line 27
   at async Task<OutputData<ImmutableArray<T>>> Pulumi.Output<T>.AllHelperAsync(ImmutableArray<Input<T>> inputs) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 288
   at async Task<(string urn, string id, Struct data, ImmutableDictionary<string, ImmutableHashSet<Resource>> dependencies)> Pulumi.Deployment.ReadOrRegisterResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 73
   at async Task<OutputData<U>> Pulumi.Output<T>.ApplyHelperAsync<U>(Task<OutputData<T>> dataTask, Func<T, Output<U>> func) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 254
   at async Task Pulumi.Deployment.CompleteResourceAsync(Resource resource, bool remote, Func<string, Resource> newDependency, ResourceArgs args, ResourceOptions options, ImmutableDictionary<string, IOutputCompletionSource> completionSources) in /_/sdk/dotnet/Pulumi/Deployment/Deployment_ReadOrRegisterResource.cs:line 89
   at async Task<T> Pulumi.Output<T>.GetValueAsync(T whenUnknown) in /_/sdk/dotnet/Pulumi/Core/Output.cs:line 153
   at async Task<string> Pulumi.Deployment+EngineLogger.TryGetResourceUrnAsync(Resource resource) in /_/sdk/dotnet/Pulumi/Deployment/Deployment.EnginerLogger.cs:line 144
Looks like I'd have to change all nested values to use 
Dictionary<string, object>
which is not desirable:
Copy code
var certManagerValues =
    new Dictionary<string, object>
    {
        ["prometheus"] = new Dictionary<string, object>
        {
            ["enabled"] = true,
            ["servicemonitor"] = new Dictionary<string, object> { ["enabled"] = true }
        }
    };
I've got heaps of nested values in other charts so this is not really practical either
This then failed with a different error:
Copy code
Diagnostics:
 
cert-manager (kubernetes:helm.sh:Release)
error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ServiceMonitor" in version "<http://monitoring.coreos.com/v1|monitoring.coreos.com/v1>"
Note: I install the CRDs separately and depend on the config group. This works with helm chart.
I think this is due to a race condition where the crds have not been created yet. I think the helm chart resources would retry and eventually succeed. Not sure if the helm release does the same.
s
@worried-city-86458 sorry missed this. Can you open an issue for this?
w
Will do
s
thanks!
4 Views
Open in Slack
PreviousNext
Powered by