https://pulumi.com logo
#general
Title
# general
b

billions-fish-49395

08/28/2023, 9:57 PM
Hey folks. I'm building an ETL pipeline with pulumi/AWS. The pipeline is meant to extract data from an S3 bucket with a Glue crawler, transform it with a Python script wrapped in a Glue job, and load the outcome into another s3 bucket. Creating the in/out buckets, secret keys, and glue role worked as expected but as soon as I wanted to create a glue catalog database, my workflow hits the timeout and fails with:
Copy code
...
...
error: 1 error occurred:
	* creating Glue Catalog Database (data-catalog-db): RequestError: send request failed
caused by: Post "<https://glue.us-east-2.amazonaws.com/>": net/http: TLS handshake timeout
error: update failed
Resources:
    12 unchanged
Duration: 58m55s
Any ideas how may I resolve this. From reading the docs, my understanding is that it has something to do with the route table and security groups that blocks the traffic but I'm very new to AWS/pulumi and I couldn’t get to the bottom of this.
b

billowy-army-68599

08/28/2023, 10:00 PM
is your glue catalog database in a private subnet? how did you deploy it?
b

billions-fish-49395

08/28/2023, 10:07 PM
Copy code
const glueDataCatalogDB = new aws.glue.CatalogDatabase(
  'data-catalog-db',
  {
     name: 'data-catalog-db',
  }
);
and it's not in a private subnet
b

billowy-army-68599

08/28/2023, 10:08 PM
hmm I’m stumped, never seen that before
b

billions-fish-49395

08/28/2023, 10:18 PM
Thanks for getting back to me! I'm getting the same error if I try to create a security config for the glue role
Copy code
const glueSecurityConfig = new aws.glue.SecurityConfiguration(
    'glue-security-config',
    {
      encryptionConfiguration: {
        s3Encryption: {
          s3EncryptionMode: 'SSE-S3',
        },
        cloudwatchEncryption: {
          cloudwatchEncryptionMode: 'SSE-KMS',
          kmsKeyArn: cloudwatchKms.arn,
        },
        jobBookmarksEncryption: {
          jobBookmarksEncryptionMode: 'CSE-KMS',
          kmsKeyArn: glueJobBookmarksKms.arn,
        },
      },
    },
    {
      dependsOn: [cloudwatchKms, glueJobBookmarksKms],
    }
  );
b

billowy-army-68599

08/28/2023, 10:18 PM
b

billions-fish-49395

08/28/2023, 10:19 PM
let me try that, thanks