Hi now i stack with a problem when the simple pulumi stack c Pulumi Community #general

Hi, now i stack with a problem, when the simple pu...

quaint-garden-96746

12/03/2019, 8:34 PM

Hi, now i stack with a problem, when the simple pulumi stack command take a lot of time

Copy code

time pulumi --logtostderr -v=9 stack
I1203 22:28:22.700826   89424 backend.go:408] found username for access token
I1203 22:29:03.110095   89424 backend.go:408] found username for access token
Current stack is stage:
    Owner: project_iac
    Last updated: 7 hours ago (2019-12-03 15:03:51.875385 +0200 EET)
    Pulumi version: v1.6.1
Current stack resources (123):
...
Current stack outputs (7):
...
real	0m40.506s
user	0m0.289s
sys	0m0.093s

the others commands like pulumi up slow as well how can i debug what is the reason of the huge slow down ?

quaint-garden-96746

12/03/2019, 10:23 PM

i tried to search soource of problem and when i did

Copy code

sudo tcpdump -i en0 "dst 54.70.73.136 || dst 35.163.58.164"

see many request to remote host during pulumi stack command runnig then search info how to trace pulumi https://www.pulumi.com/docs/troubleshooting/ and found this:

quaint-garden-96746

12/03/2019, 10:29 PM

it generates 193 requests to api

quaint-garden-96746

12/03/2019, 10:31 PM

colossal-beach-47527

12/04/2019, 4:25 PM

Could you please file an issue on github.com/pulumi/pulumi? It would be useful to know the general shape of your Pulumi program’s configuration and resources. (i.e. what would cause those decryptValue calls in the first place.) We have some optimizations in place to avoid redundant calls to decrypt values, but it could be that you are hitting some codepath that isn’t using cached/memoized values. /cc @microscopic-florist-22719

👍 1

quaint-garden-96746

12/04/2019, 8:57 PM

thx you for the advice, seams i get the source of the problem. This is production ready env that contains many k8s.apps.v1.Deployment that spec became secrets coz its use k8s.core.v1.Secret as input for .envFrom.secretRef also i have k8s.core.v1.Service that also became secrets coz its use deployment.spec.template.metadata.labels as input for selector even all k8s.networking.v1beta1.Ingress became secret coz its use service.metadata.name as input for serviceName

🙏🏽 1

quaint-garden-96746

12/04/2019, 9:03 PM

@colossal-beach-47527 i’ll try to test and collect more information and then I’ll create an issues.

quaint-garden-96746

12/04/2019, 11:22 PM

finnaly i found the true source or the problem

Copy code

secret           : {
        apiVersion: "v1"
        data      : "[secret]"
        id        : "default/backend-api-secrets-ahdxt3ym"
        kind      : "Secret"
        metadata  : "[secret]"
        stringData: "[secret]"
        type      : "Opaque"
        urn       : "urn:pulumi:stage::project-aws-eks::k8s:app:backend$kubernetes:core/v1:Secret::backend-api-secrets"
    }

metadata of the k8s.core.v1.Secret became secret I use it for the creating service account with providing registry secret with secret.metadata.name but it’s encrypted and my service account became encrypted as well and then I use sa.metadata.name for my deployment and its also became encrypted all my k8s resources became encrypted 😄. i found workaround with:

Copy code

secret.id.apply(v => v.replace(/^[^\/]+\//,''))

It’s not a good solution at all. I think not all metadata in secrets must be encrypted -> name at least

quaint-garden-96746

12/04/2019, 11:42 PM

now pulumi stack run done in ~10 sec

quaint-garden-96746

12/04/2019, 11:42 PM

4 Views

Open in Slack

Previous Next