Hey folks. I'm building an ETL pipeline with pulumi/AWS. The pipeline is meant to extract data from an S3 bucket with a Glue crawler, transform it with a Python script wrapped in a Glue job, and load the outcome into another s3 bucket. Creating the in/out buckets, secret keys, and glue role worked as expected but as soon as I wanted to create a glue catalog database, my workflow hits the timeout and fails with:

...
...
error: 1 error occurred:
	* creating Glue Catalog Database (data-catalog-db): RequestError: send request failed
caused by: Post "<https://glue.us-east-2.amazonaws.com/>": net/http: TLS handshake timeout
error: update failed
Resources:
    12 unchanged
Duration: 58m55s

Any ideas how may I resolve this. From reading the docs, my understanding is that it has something to do with the route table and security groups that blocks the traffic but I'm very new to AWS/pulumi and I couldn’t get to the bottom of this.

is your glue catalog database in a private subnet? how did you deploy it?

const glueDataCatalogDB = new aws.glue.CatalogDatabase(
  'data-catalog-db',
  {
     name: 'data-catalog-db',
  }
);

and it's not in a private subnet

hmm I’m stumped, never seen that before

Thanks for getting back to me! I'm getting the same error if I try to create a security config for the glue role

const glueSecurityConfig = new aws.glue.SecurityConfiguration(
    'glue-security-config',
    {
      encryptionConfiguration: {
        s3Encryption: {
          s3EncryptionMode: 'SSE-S3',
        },
        cloudwatchEncryption: {
          cloudwatchEncryptionMode: 'SSE-KMS',
          kmsKeyArn: cloudwatchKms.arn,
        },
        jobBookmarksEncryption: {
          jobBookmarksEncryptionMode: 'CSE-KMS',
          kmsKeyArn: glueJobBookmarksKms.arn,
        },
      },
    },
    {
      dependsOn: [cloudwatchKms, glueJobBookmarksKms],
    }
  );

can you try verbose logs? https://www.pulumi.com/docs/support/troubleshooting/#verbose-logging

let me try that, thanks