How does Pulumi on AWS with Databricks interact? https://docs.databricks.com/en/dev-tools/pulumi.html mentions that the workspace must already be deployed/created. But I was hoping to create the databricks worksapces automatically from pulumi. How would this work instead?

You can definitely do it, it’s this resource: https://www.pulumi.com/registry/packages/databricks/api-docs/mwsworkspaces/

thanks

nope, but if you let me know what you’re trying to do I can maybe help

Thanks! This would be awesome. I have 3 AWS accounts dev/test/prod and want to create a databricks workspace in all 3 stacks. Then I want to create some autoscaling cluster in the workspace. Furthermore, I want to create some S3 buckets a) one for my data outputs b) any necessary ones for the databricks workspace to live happily in their account. Any useful information like URLs to sign in or bucket ids/names should be in the stack readme (https://www.pulumi.com/docs/pulumi-cloud/projects-and-stacks/?_ga=2.149016052.318359768.1694094860-180507132.1691596035#stack-readme). For the reademe: I was wollowing https://www.pulumi.com/docs/pulumi-cloud/projects-and-stacks/?_ga=2.149016052.318359768.1694094860-180507132.1691596035#stack-readme

import pulumi
from pulumi_aws_native import s3

# Create an AWS resource (S3 Bucket)
bucket = s3.Bucket("my_bucket")

pulumi.export("my_bucket", bucket.id)

with open('./Pulumi.README.md') as f:
    pulumi.export('readme', f.read())

The readme with contents of:

- main bucket ${bucket.id}
- my_bucket ${my_bucket}

Is neatly uploaded, however the ${} reference to the variable is never resolved and empty For the databricks workspace:

workspace = mws.Workspaces("my_workspace", 
    account_id="<AccountId>", # Your Account Id
    aws_region="<AWSRegion>",  # The AWS region for the VPC
    credentials_id="<CredentialsId>",  # Your credential id
    storage_configuration_id="<StorageConfigurationId>",  # Your storage configuration id
    network_id="<NetworkId>",  # Your network id
    workspace_name="<WorkSpaceName>",)  # Name for your workspace

- how should I retrieve the AWS region? - https://www.pulumi.com/docs/concepts/config/ - should I us the require concept:

name = config.require('name');

? - or can these be auto-resolved from a configured AWS CLI? - how to set the following: - credentials_id - storage_configuration_id - network_id I think these might require setting up some more S3 buckets or VPCs for databricks? But am unsure what to put there. Regarding the cluster: I think https://docs.databricks.com/en/dev-tools/pulumi.html is already describing how to set up the cluster. But if you could include a mini dummy cluster example this would be very helpful as well.

unfortunately databricks is not my area of expertise.

yes

I have a lot of AWS experience but no databricks experience I’m afraid,

Understood - I will try to go step-by-step for https://docs.databricks.com/en/administration-guide/account-settings-e2/credentials.html#step-1-create-a-cross-account-iam-role and turning each individual step into a pulumi automation script.