https://pulumi.com logo
Powered by
Title
e

echoing-smartphone-60420

04/12/2022, 1:10 PM
Hmm, still fighting my Helm template issue here 🧵. Working for helm but not 
.release
api
So I'm trying to run this: https://github.com/temporalio/helm-charts
confirmed with basic 
values.yml
that will build in about 3 mins running with 
helm install/upgrade
but running as part of Pulumi it freezes for about 5 minutes then timesout
I can see resources being created by pulumi but they get rolled back by the 
--atomic
flag at the end of it
There are subcharts within here that have been preinstalled with 
helm dependencies update
pulumi logs
and 
pulumi up v=9
doesn't yield anything usefull
could it be that I just need to add more time to the deployment timeout?
---
trying a run now with 
skipAwait
and 
timeout 6000
const temporal = new k8s.helm.v3.Release("temp", {
    chart: "./temporal",
    version: "1.0.0",
    atomic: true,
    skipAwait: true,
    timeout: 6000,
    values: {
      server: {
        replicaCount:1
      },
      cassandra: {
        config: {
          cluster_size: 1,
        }
      },
      prometheus: {
        enabled: true
      },
      grafana: { enabled: true},
      elasticearch: { enabled: false},
    }
});
g

great-queen-39697

04/13/2022, 3:21 PM
Hi Zach! Good to see you on here! Regarding the log situation, I'm in the middle of fixing those docs. Short version is the verbose flag only gives you logs from the engine, not the provider. You'll need to add different flags to get output to bubble up from those. Overall, the exact flags that you want depends on whether any providers you're using are using the Terraform bridge. But here's a short snippet that should get you all of the diagnostic data that's possible if you can run it locally: 
TF_LOG=TRACE pulumi up -v=11 --logflow --logtostderr 2>&1 | tee -a pulumi_log.txt
The environment variable asks the classic provider to bubble up data. The 
--logflow
flag bubbles up data from providers that aren't on the Terraform bridge.
It looks like you solved the main problem you were running into, though?
e

echoing-smartphone-60420

04/13/2022, 4:33 PM
Thanks for logging, I'll give that shot and see if I can get more info. I'm able to launch operations via the helm.release api however it seems to be getting stuck along the way. What takes about 3mins on helm sits running on pulumi deploy until a sigint
or timeout
let me see if I get anything out of the increased verbosity here
My current theory that it has to do with one of the helm hooks/dependencies/subcharts being used by the temporal chart
g

great-queen-39697

04/13/2022, 4:35 PM
ahh, got it. If you want to dump your output for me (and redact/send me a private gist if there's confidential stuff in there), I'll go digging if you don't find a good bit of info
e

echoing-smartphone-60420

04/13/2022, 4:38 PM
:thx:
building the stack now...
hmm interesting, running with the increased verbosity looks to panic on some iam Userpolicies being generated:
I'll run this normally, then turn on the logging just for the helm bits
looks to be related to https://github.com/pulumi/pulumi/issues/6583
hmm, I had some older versions of the providers loaded ,but upgrading them here didn't seem to help.
b

billowy-army-68599

04/14/2022, 11:30 PM
Set a reminder to come back to this tomorrow
:thank-you: 1
g

great-queen-39697

04/14/2022, 11:32 PM
@echoing-smartphone-60420, Lee's been here longer and knows way more than me about our Helm stuff as I'm still learning it. We'll figure this out :)
e

echoing-smartphone-60420

04/15/2022, 1:33 AM
Thanks. We're including this a next gen tech stack proposal for a client shipping here in the next month or so. So we've got a little time. Lmk if you want to jump on a session to kick tires.
b

billowy-army-68599

04/15/2022, 1:38 AM
I'm in transit tomorrow, should definitely be able to help on Monday
@echoing-smartphone-60420 following up here: when you're running the helm relese from pulumi and it gets stuck, what's actually happening in the cluster?
e

echoing-smartphone-60420

04/15/2022, 5:38 PM
im seeing some resources provision from kubectl but not seeing a release installed on the helm cli
let me see where its getting stuck
b

billowy-army-68599

04/15/2022, 5:41 PM
I can try repro when I'm not flying, the local helm chart you're using is the same as the upstream one?
e

echoing-smartphone-60420

04/15/2022, 5:53 PM
alright I got the infra up
I'll run the release resource here with the log leve
Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:main::grand-infra::pulumi:pulumi:Stack::grand-infra-main]
    + kubernetes:<http://helm.sh/v3:Release|helm.sh/v3:Release>: (create)
        [urn=urn:pulumi:main::grand-infra::kubernetes:<http://helm.sh/v3:Release::temp|helm.sh/v3:Release::temp>]
        [provider=urn:pulumi:main::grand-infra::pulumi:providers:kubernetes::default_3_18_2::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        atomic                  : true
        chart                   : "./temporal"
        cleanupOnFail           : false
        createNamespace         : false
        dependencyUpdate        : false
        devel                   : false
        disableCRDHooks         : false
        disableOpenapiValidation: false
        disableWebhooks         : false
        forceUpdate             : false
        lint                    : false
        name                    : "temp-b1f36b40"
        namespace               : "default"
        recreatePods            : false
        renderSubchartNotes     : false
        replace                 : false
        resetValues             : false
        resourceNames           : {
            <http://ClusterRole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1|ClusterRole.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1>       : [
                [0]: "temp-b1f36b40-kube-state-metrics"
                [1]: "temp-b1f36b40-prometheus-alertmanager"
                [2]: "temp-b1f36b40-prometheus-pushgateway"
                [3]: "temp-b1f36b40-prometheus-server"
            ]
            <http://ClusterRoleBinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1|ClusterRoleBinding.rbac.authorization.k8s.io/rbac.authorization.k8s.io/v1>: [
                [0]: "temp-b1f36b40-kube-state-metrics"
                [1]: "temp-b1f36b40-prometheus-alertmanager"
                [2]: "temp-b1f36b40-prometheus-pushgateway"
                [3]: "temp-b1f36b40-prometheus-server"
            ]
            ConfigMap/v1                                                             : [
                [0]: "default/temp-b1f36b40-grafana"
                [1]: "default/temp-b1f36b40-grafana-dashboards-default"
                [2]: "default/temp-b1f36b40-prometheus-alertmanager"
                [3]: "default/temp-b1f36b40-prometheus-server"
                [4]: "temp-b1f36b40-temporal-dynamic-config"
                [5]: "temp-b1f36b40-temporal-frontend-config"
                [6]: "temp-b1f36b40-temporal-history-config"
                [7]: "temp-b1f36b40-temporal-matching-config"
                [8]: "temp-b1f36b40-temporal-web-config"
                [9]: "temp-b1f36b40-temporal-worker-config"
            ]
            Deployment.apps/apps/v1                                                  : [
                [0]: "default/temp-b1f36b40-grafana"
                [1]: "default/temp-b1f36b40-kube-state-metrics"
                [2]: "default/temp-b1f36b40-prometheus-alertmanager"
                [3]: "default/temp-b1f36b40-prometheus-pushgateway"
                [4]: "default/temp-b1f36b40-prometheus-server"
                [5]: "temp-b1f36b40-temporal-admintools"
                [6]: "temp-b1f36b40-temporal-frontend"
                [7]: "temp-b1f36b40-temporal-history"
                [8]: "temp-b1f36b40-temporal-matching"
                [9]: "temp-b1f36b40-temporal-web"
                [10]: "temp-b1f36b40-temporal-worker"
            ]
            Job.batch/batch/v1                                                       : [
                [0]: "temp-b1f36b40-temporal-es-index-setup"
                [1]: "temp-b1f36b40-temporal-schema-setup"
                [2]: "temp-b1f36b40-temporal-schema-update"
            ]
            PersistentVolumeClaim/v1                                                 : [
                [0]: "default/temp-b1f36b40-prometheus-alertmanager"
                [1]: "default/temp-b1f36b40-prometheus-server"
            ]
            Pod/v1                                                                   : [
                [0]: "temp-b1f36b40-wismc-test"
            ]
            PodDisruptionBudget.policy/policy/v1beta1                                : [
                [0]: "elasticsearch-master-pdb"
            ]
            Secret/v1                                                                : [
                [0]: "default/temp-b1f36b40-grafana"
                [1]: "temp-b1f36b40-temporal-default-store"
                [2]: "temp-b1f36b40-temporal-visibility-store"
            ]
            Service/v1                                                               : [
                [0]: "default/temp-b1f36b40-grafana"
                [1]: "default/temp-b1f36b40-kube-state-metrics"
                [2]: "default/temp-b1f36b40-prometheus-alertmanager"
                [3]: "default/temp-b1f36b40-prometheus-pushgateway"
                [4]: "default/temp-b1f36b40-prometheus-server"
                [5]: "elasticsearch-master"
                [6]: "elasticsearch-master-headless"
                [7]: "temp-b1f36b40-cassandra"
                [8]: "temp-b1f36b40-temporal-admintools"
                [9]: "temp-b1f36b40-temporal-frontend"
                [10]: "temp-b1f36b40-temporal-frontend-headless"
                [11]: "temp-b1f36b40-temporal-history-headless"
                [12]: "temp-b1f36b40-temporal-matching-headless"
                [13]: "temp-b1f36b40-temporal-web"
                [14]: "temp-b1f36b40-temporal-worker-headless"
            ]
            ServiceAccount/v1                                                        : [
                [0]: "default/temp-b1f36b40-grafana"
                [1]: "default/temp-b1f36b40-kube-state-metrics"
                [2]: "default/temp-b1f36b40-prometheus-alertmanager"
                [3]: "default/temp-b1f36b40-prometheus-pushgateway"
                [4]: "default/temp-b1f36b40-prometheus-server"
                [5]: "temporaladmin"
            ]
            StatefulSet.apps/apps/v1                                                 : [
                [0]: "elasticsearch-master"
                [1]: "temp-b1f36b40-cassandra"
            ]
        }
        reuseValues             : false
        skipAwait               : true
        skipCrds                : false
        timeout                 : 6000
        values                  : {
            cassandra   : {
                config: {
                    cluster_size: 1
                }
            }
            elasticearch: {
                enabled: false
            }
            grafana     : {
                enabled: true
            }
            prometheus  : {
                enabled: true
            }
            server      : {
                replicaCount: 1
            }
        }
        verify                  : false
        version                 : "1.0.0"
        waitForJobs             : false
        ~ pulumi:providers:kubernetes: (update)
            [id=b62bfb25-6546-424a-92b1-384e58dfcc80]
            [urn=urn:pulumi:main::grand-infra::eks:index:Cluster$pulumi:providers:kubernetes::grand-eks-eks-k8s]
          ~ version: "3.17.0" => "3.18.2"
        ~ pulumi:providers:kubernetes: (update)
            [id=b69eea56-b079-4318-8853-c6f2cbd7e146]
            [urn=urn:pulumi:main::grand-infra::eks:index:Cluster$pulumi:providers:kubernetes::grand-eks-provider]
          ~ version: "3.17.0" => "3.18.2"
☝️ there's the 
.Values
its a super vanila deploy
trying now with 3.18.2 to as well
https://gist.github.com/bevel-zgates/3d0135f7ae200e8d1d13dcd2044bb15c
b

billowy-army-68599

04/18/2022, 4:08 PM
alright I got the infra up
@echoing-smartphone-60420 does this mean you got it running? anything else you need help debugging with here?
e

echoing-smartphone-60420

04/18/2022, 6:02 PM
No I was meaning the cluster itself.
After upgrading to 3.18.2 I'm still running into an error when trying to run the .release
b

billowy-army-68599

04/18/2022, 6:25 PM
okay, trying to deploy it myself now
@echoing-smartphone-60420 one thing I noticed: • you aren't passing a provider to the helm template at all, so where is the helm chart going?
also, did you pull the sub charts locally? I'm having a hard time getting those running
e

echoing-smartphone-60420

04/18/2022, 6:45 PM
ah, that's a good point
let me take a look here
I also did pull the subcharts in
b

billowy-army-68599

04/18/2022, 6:46 PM
yeah I just ran 
helm dependency update
and it pulled them
e

echoing-smartphone-60420

04/18/2022, 6:53 PM
interesting, I didn't even see the k8s.provider, but I assume its using the default or active context from kubeconfig
b

billowy-army-68599

04/18/2022, 6:53 PM
yeah, it uses the "ambient" provider
e

echoing-smartphone-60420

04/18/2022, 6:53 PM
Though not sure how it knows what namespace to use
guess default
b

billowy-army-68599

04/18/2022, 6:53 PM
you can disable it on the stack to prevent it doing weird things
yeah it'll use default
e

echoing-smartphone-60420

04/18/2022, 6:59 PM
So to answer your question: 
@Zach Gates one thing I noticed:
you aren't passing a provider to the helm template at all, so where is the helm chart going?
I'm running 
aws eks update-kubeconfig --name <name of cluster>
after completing the cluster creation earlier in script
b

billowy-army-68599

04/18/2022, 7:00 PM
got it, so it's going to the right place
e

echoing-smartphone-60420

04/18/2022, 7:01 PM
probably not relevant but i'm using 
aws-vault
to wrap the pulumi call with aws-environment variables
b

billowy-army-68599

04/18/2022, 7:57 PM
@echoing-smartphone-60420 this provisioned correctly for me: https://gist.github.com/jaxxstorm/8880143ed115e95b751694397b1cdbbe Note, I added a 
dependsOn
for the helm chart to the node group, what I found was that the pods were stuck pending 
NAME                                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                                   0/1     Pending   0          38m
elasticsearch-master-1                                   0/1     Pending   0          38m
elasticsearch-master-2                                   0/1     Pending   0          38m
temp-dd4cb578-cassandra-0                                0/1     Pending   0          38m
temp-dd4cb578-grafana-dd58f468d-d2pg5                    0/1     Pending   0          38m
temp-dd4cb578-kube-state-metrics-548559c9cd-c6m9s        0/1     Pending   0          38m
temp-dd4cb578-prometheus-alertmanager-6b4b6cb5b7-m5vv8   0/2     Pending   0          38m
temp-dd4cb578-prometheus-pushgateway-b7d4f96d6-tvm7z     0/1     Pending   0          38m
temp-dd4cb578-prometheus-server-5d9674bd4c-qs4vg         0/2     Pending   0          38m
temp-dd4cb578-temporal-admintools-6b8bf64cb5-7zhbs       0/1     Pending   0          38m
temp-dd4cb578-temporal-frontend-5db9b4d79-6ld72          0/1     Pending   0          38m
temp-dd4cb578-temporal-history-666dc56744-jgqnn          0/1     Pending   0          38m
temp-dd4cb578-temporal-matching-5d468745c6-s8t9l         0/1     Pending   0          38m
temp-dd4cb578-temporal-web-7b77d477c8-gllhf              0/1     Pending   0          38m
temp-dd4cb578-temporal-worker-758b668454-tl6pp           0/1     Pending   0          38m
the nodegroups takes a little while to provision because of the fargate/spot bid, which then the helm release times out
adding a dependsOn waits for the workload to provision, and then starts the helm release
🎉 1
I ran it twice and it worked both times, but it takes between 20m/30m
my last attempt failed because the spot bid wasn't successful 😄
e

echoing-smartphone-60420

04/18/2022, 8:46 PM
ah yes
sweet, lemme give it a shot here
appreciate your time/effort
84 Views
#getting-startedJoin Slack
Powered by