https://pulumi.com logo
Powered by
Title
q

quiet-leather-94755

01/27/2021, 7:02 AM
Hi! 👋 I'm having some issues with replacing an existing aws.servicediscovery.PrivateDnsNamespace().. it's trying to delete the namespace, but there are still ECS / Fargate services running that use the old namespace (see thread for output). Seems like a bug, perhaps?
Output from `pulumi up`:
Previewing update (dev)

View Live: <https://app.pulumi.com/sch_johwar2/redata-aws-fargate/dev/previews/805ae9ef-3aae-402e-aa38-0e3902c9cb26>

     Type                                         Name                      Plan        Info
     pulumi:pulumi:Stack                          redata-aws-fargate-dev                
 +-  ├─ aws:ecs:TaskDefinition                    airflow-web-task          replace     
 +-  ├─ aws:ecs:TaskDefinition                    airflow-scheduler-task    replace     
 +-  ├─ aws:ecs:TaskDefinition                    grafana-web-task          replace     
 +-  ├─ aws:servicediscovery:PrivateDnsNamespace  redata-sd-namespace       replace     [diff: ~name]
 ~   ├─ aws:efs:FileSystemPolicy                  redata-efs-policy         update      [diff: ~policy]
     ├─ redata:service:BackendService             airflow-scheduler                     
 +-  │  ├─ aws:ecs:Service                        airflow-scheduler-svc     replace     [diff: ~name,serviceRegistries,taskDefinition]
 +-  │  └─ aws:servicediscovery:Service           airflow-scheduler-sd-svc  replace     [diff: ~dnsConfig]
     ├─ redata:service:WebService                 airflow-web                           
     │  └─ redata:service:BackendService          airflow-web                           
 +-  │     ├─ aws:ecs:Service                     airflow-web-svc           replace     [diff: ~name,serviceRegistries,taskDefinition]
 +-  │     └─ aws:servicediscovery:Service        airflow-web-sd-svc        replace     [diff: ~dnsConfig]
     ├─ redata:service:WebService                 grafana-web                           
     │  └─ redata:service:BackendService          grafana-web                           
 +-  │     ├─ aws:ecs:Service                     grafana-web-svc           replace     [diff: ~name,serviceRegistries,taskDefinition]
 +-  │     └─ aws:servicediscovery:Service        grafana-web-sd-svc        replace     [diff: ~dnsConfig]
 +   ├─ aws:route53:Record                        airflow-postgres-cname    create      
 +   └─ aws:route53:Record                        redata-postgres-cname     create      
 
Resources:
    + 2 to create
    ~ 1 to update
    +-10 to replace
    13 changes. 33 unchanged

Do you want to perform this update? yes
Updating (dev)

View Live: <https://app.pulumi.com/sch_johwar2/redata-aws-fargate/dev/updates/119>

     Type                                         Name                    Status                   Info
     pulumi:pulumi:Stack                          redata-aws-fargate-dev  **failed**               1 error
 +-  ├─ aws:ecs:TaskDefinition                    airflow-web-task        replaced                 
 ~   ├─ aws:efs:FileSystemPolicy                  redata-efs-policy       updated                  [diff: ~policy]
 +-  ├─ aws:ecs:TaskDefinition                    airflow-scheduler-task  replaced                 
 +-  ├─ aws:ecs:TaskDefinition                    grafana-web-task        replaced                 
 +-  └─ aws:servicediscovery:PrivateDnsNamespace  redata-sd-namespace     **replacing failed**     1 error
 
Diagnostics:
  pulumi:pulumi:Stack (redata-aws-fargate-dev):
    error: update failed
 
  aws:servicediscovery:PrivateDnsNamespace (redata-sd-namespace):
    error: deleting urn:pulumi:dev::redata-aws-fargate::aws:servicediscovery/privateDnsNamespace:PrivateDnsNamespace::redata-sd-namespace: 1 error occurred:
    	* error deleting Service Discovery Private DNS Namespace (ns-o4ilckypsuym75iu): ResourceInUse: Namespace has associated services; delete the services before deleting the namespace
 
Outputs:
    airflow-db         : "<http://airflow-postgresacfea51.co25doitymly.eu-west-1.rds.amazonaws.com:5432|airflow-postgresacfea51.co25doitymly.eu-west-1.rds.amazonaws.com:5432>"
    airflow-web-web-url: "<https://redata.blocket-insight-dev.schibsted.io/airflow>"
    grafana-web-web-url: "<https://redata.blocket-insight-dev.schibsted.io/grafana>"
    redata-db          : "<http://redata-postgres36ddbfa.co25doitymly.eu-west-1.rds.amazonaws.com:5432|redata-postgres36ddbfa.co25doitymly.eu-west-1.rds.amazonaws.com:5432>"

Resources:
    ~ 1 updated
    33 unchanged

Duration: 11s
The changes I'm trying to make: 
diff --git a/deployment/pulumi_aws_fargate/__main__.py b/deployment/pulumi_aws_fargate/__main__.py
index 01fa5a3..a2ed3e4 100644
--- a/deployment/pulumi_aws_fargate/__main__.py
+++ b/deployment/pulumi_aws_fargate/__main__.py
@@ -127,6 +127,7 @@ efs = redata.fs.FileSystem("redata-efs",
 
 # Service Discovery for intra-cluster communication
 sd_namespace = aws.servicediscovery.PrivateDnsNamespace("redata-sd-namespace",
+    name="redata.local",
     description="Private namespace for Redata services",
     vpc=redata_vpc.id,
 )
@@ -194,6 +195,13 @@ for backend in ("airflow", "redata"):
         username=backend,
         vpc_security_group_ids=[db_secgrp.id]
     )
+    db_cname = aws.route53.Record(f"{backend}-postgres-cname",
+        name=f"{backend}-postgres",
+        zone_id=sd_namespace.hosted_zone,
+        type='CNAME',
+        ttl=60,
+        records=[db.address],
+    )
     databases[backend] = db
     export(f"{backend}-db", db.endpoint)
Link to full version of code, before the above changes: https://github.com/redata-team/redata/pull/31
f

faint-table-42725

01/27/2021, 7:11 PM
I’m a bit confused on how the replacement is happening? I don’t see the namespace in the PR you linked.
It looks like what’s checked in has “redata-sd-local-namespace” which is different from the diff you have above. Were you able to get this working?
q

quiet-leather-94755

01/27/2021, 7:45 PM
Yeah, I had to add the modified namespace first, with a different name.. Then after 'pulumi up' replaced all the references, I could remove the old version.
The issue seems to be that while Pulumi does replace references in the service definitions, it's not waiting for the old tasks to be fully phased out (which takes a few minutes).
I, uh, did a lot of changes and then rebased + force pushed, so the link isn't that useful now 😅
f

faint-table-42725

01/27/2021, 10:50 PM
No worries — just wanted to make sure you got unblocked here. Regarding the general use case, you may want to set 
wait_for_steady_state
on the 
ecs.Service
This is similar to https://docs.aws.amazon.com/cli/latest/reference/ecs/wait/services-stable.html
q

quiet-leather-94755

01/28/2021, 5:43 AM
Ahh ok, thanks! 👍
3 Views
#awsJoin Slack
Powered by