https://pulumi.com logo
Powered by
n

numerous-eye-91601

09/06/2023, 7:50 PM
anybody else having issues managing more than one Kubernetes cluster under one Pulumi stack? Primarily as a result of ConfigFile and helm Chart giving 
try giving it a unique name
seems like the only way it gets read in is via the k8s object name but I want them to be the same. in some cases they can't be changed anyways like for CRDs
b

billowy-army-68599

09/06/2023, 8:14 PM
can you share your code?
n

numerous-eye-91601

09/06/2023, 8:18 PM
an example using the var 
cluster_name
to inject the specific eks cluster name
Copy code
eso = Chart(
    f"{cluster_name}-eso-chart",
    config=ChartOpts(
        chart="external-secrets",
        version="0.9.2",
        namespace="external-secrets",
        fetch_opts=FetchOpts(
            repo="<https://charts.external-secrets.io>",
        ),
        values={"fullnameOverride": "eso"},
    ),
    opts=pulumi.ResourceOptions(
        depends_on=[namespace],
        provider=self.cluster_provider,
        deleted_with=namespace,
    ),
)
b

billowy-army-68599

09/06/2023, 8:19 PM
and cluster name is unique?
n

numerous-eye-91601

09/06/2023, 8:19 PM
error related:
Copy code
Diagnostics:
  kubernetes:core/v1:ServiceAccount (external-secrets/external-secrets-cert-controller):
    error: Duplicate resource URN 'urn:pulumi:tst::aws-eks::kubernetes:<http://helm.sh/v3:Chart$kubernetes:core/v1:ServiceAccount::external-secrets/external-secrets-cert-controller';|helm.sh/v3:Chart$kubernetes:core/v1:ServiceAccount::external-secrets/external-secrets-cert-controller';> try giving it a unique name
yeah, it seems the subresources helm pops out are not though
b

billowy-army-68599

09/06/2023, 8:23 PM
this seems like an issue in the chart. Generally, the chart best practice is to set the name to something related to the releasename. External secrets doesn’t seem to do that: https://github.com/external-secrets/external-secrets/blob/main/deploy/charts/external-secrets/templates/_helpers.tpl#L130 So the release name isn’t being interpolated and creating duplicate resources. I’d suggest overriding the resources to st the name property in the values using the cluster name
or, you might choose to use 
helm.Release
instead, which doesn’t suffer from this issue
n

numerous-eye-91601

09/06/2023, 8:26 PM
hm, yeah I've been switching to helm.Release to work around it some. in some cases it's not ideal to have to rename everything with that prefix. for example I am using ConfigFile to load in a 
ClusterSecretStore
for ESO but having that renamed to specifically prefix the cluster name messes up a lot of things that use that in the codebase
sounds like there is no other way to force those subresources to have a unique URN then? would have to either rename or use a different stack?
b

billowy-army-68599

09/06/2023, 8:27 PM
but not sure if you can manipulate the urn from there
n

numerous-eye-91601

09/06/2023, 8:29 PM
hm, yeah not sure either. used that previously to rename the name but will poke around and see if urn renaming is possible there instead
doesn't seem like the transform will work unfortunately to do that, 
ResourceTransformationResult
only deals with inputs and outputs, not internal attributes. tried to hack around that but no success. thanks anyways though!
for anyone stumbling onto this after I was able to work around the issue by relying on Release instead of Chart and also by turning all my other resources into CustomResources. this is not the most elegant but I just needed to get argocd setup with ESO so it wasn't too bad for me. You could probably create some boilerplate code to make it easier to create commonly used crds. I'm including a sample ClusterSecretStore example resource below for those interested:
Copy code
secretstore = CustomResource(
    f"{cluster_name}-doppler-secret-store",
    api_version="<http://external-secrets.io/v1beta1|external-secrets.io/v1beta1>",
    kind="ClusterSecretStore",
    metadata=ObjectMetaArgs(
        name="doppler-auth",
        namespace="external-secrets",
    ),
    spec={
        "provider": {
            "doppler": {
                "auth": {
                    "secretRef": {
                        "dopplerToken": {
                            "name": "doppler-secret-token",
                            "key": "dopplerToken",
                            "namespace": "external-secrets"
                        }
                    }
                },
                "project": doppler_project,
                "config": doppler_config
            }
        }
    },
    opts=pulumi.ResourceOptions(
        depends_on=[eso, doppler_secret_token],
        provider=self.cluster_provider,
    )
)
3 Views
Powered by