https://pulumi.com logo
Join Slack
Powered by
After upgrading to Pulumi v3.122.0, `pulumi up` ta...
# general
d
After upgrading to Pulumi v3.122.0, 
pulumi up
takes a LONG time (>20 minutes, even when making relatively few updates) with a Google Cloud Storage Bucket as the backend, also there is a LOT (hundreds of megabytes, to gigabytes) of traffic to Google Cloud during that time. The stack JSON as a set of bytes is ~3MiB, so it makes no sense that it would be doing THAT much I/O. Running 
pulumi refresh
(with or without 
--skip-preview
) is unaffected. Is this a known issue ? I wasn't able to really find anything documented on it. One thing that did come up was checkpoints, but during the time that excessive network I/O is happening no resources are being updated. Experiments to try: 1. Disable checkpointing 2. Use tracing
Example run:
Copy code
rkeene@laptop$ pulumi up --skip-preview 
Updating (staging):
     Type                                       Name                                     Status             Info
     pulumi:pulumi:Stack                        compx-consumer-production-cloud-staging                     1 warning; 2 messages
 ~   ├─ gcp:monitoring:Dashboard                dashboard                                updated (3s)       [diff: ~dashboardJson]
 ~   ├─ gcp:compute:Firewall                    system-peer-mexico-vpn-nat-ssh           updated (72s)      [diff: ~priority]
     ├─ gcp:compute:Network                     system-peer-mexico-vpn-network                              
     │  ├─ gcp:compute:Subnetwork               system-peer-mexico-vpn-subnet-us-west2                      
 ~   │  │  └─ gcp:compute:Instance              system-peer-mexico-vpn-natvm             updated (103s)     [diff: ~metadata]
 +   │  ├─ gcp:compute:Firewall                 peer-mexico-webhook-server-via-2         created (25s)      
 +   │  ├─ gcp:compute:Firewall                 peer-mexico-webhook-server-via-1         created (20s)      
 -   │  └─ gcp:compute:Firewall                 peer-mexico-webhook-server-via           deleted (14s)      
     ├─ Compx:GCP:LegacyBankingService          system-app-legacypeering                                    
 ~   │  └─ gcp:cloudrun:Service                 system-app-legacypeering-us-west2-crs    updated (4s)       [diff: ~metadata,template]
     └─ Compx:GCP:SystemNetworking              system-network                                              
        └─ Compx:GCP:SystemDB                   system-db                                                   
           └─ Compx:GCP:SystemApp               system-app                                                  
              ├─ Compx:GCP:SystemPortalRegion   system-app-portal-us-west2                                  
 ~            │  └─ gcp:cloudrun:Service        system-app-portal-us-west2-service       updated (6s)       [diff: ~metadata,template]
              ├─ Compx:GCP:CSSM                 system-app-cssm                                             
              │  └─ gcp:serviceAccount:Account  system-app-cssm-service-account                             
 ~            │     └─ gcp:cloudrun:Service     system-app-cssm-cr                       updated (6s)       [diff: ~metadata,template]
              └─ Compx:GCP:SystemAPIRegion      system-app-api-us-west2                                     
 ~               └─ gcp:cloudrun:Service        system-app-api-us-west2-service          updated (3s)       [diff: ~metadata,template]

Diagnostics:
  pulumi:pulumi:Stack (compx-consumer-production-cloud-staging):
    Provider not given to SystemDocs, this may cause issues with GoogleCloudFolder

    warning: Undefined value (default) will not show as a stack output.

    Using Managed GCP DNS Zone: compxnet-dns-d69938b

Resources:
    + 2 created
    ~ 7 updated
    - 1 deleted
    10 changes. 749 unchanged

Duration: 23m22s
Duration is 23m, but updating time is 4min 16s, meaning 19min 6s was spent doing "nothing", during that "nothing" >300 MiB of I/O was transferred
The stack "runtime" (i.e., computing the changes) finishes within 10 seconds
With 
PULUMI_SKIP_CHECKPOINTS=1
, the time for a change (with 6 updates ) was 23seconds
40 Views
Open in Slack
PreviousNext
Powered by