Thanks! This would be awesome.
I have 3 AWS accounts dev/test/prod and want to create a databricks workspace in all 3 stacks. Then I want to create some autoscaling cluster in the workspace. Furthermore, I want to create some S3 buckets a) one for my data outputs b) any necessary ones for the databricks workspace to live happily in their account. Any useful information like URLs to sign in or bucket ids/names should be in the stack readme (
https://www.pulumi.com/docs/pulumi-cloud/projects-and-stacks/?_ga=2.149016052.318359768.1694094860-180507132.1691596035#stack-readme).
For the reademe: I was wollowing
https://www.pulumi.com/docs/pulumi-cloud/projects-and-stacks/?_ga=2.149016052.318359768.1694094860-180507132.1691596035#stack-readme
import pulumi
from pulumi_aws_native import s3
# Create an AWS resource (S3 Bucket)
bucket = s3.Bucket("my_bucket")
pulumi.export("my_bucket", bucket.id)
with open('./Pulumi.README.md') as f:
pulumi.export('readme', f.read())
The readme with contents of:
- main bucket ${bucket.id}
- my_bucket ${my_bucket}
Is neatly uploaded, however the ${} reference to the variable is never resolved and empty
For the databricks workspace:
workspace = mws.Workspaces("my_workspace",
account_id="<AccountId>", # Your Account Id
aws_region="<AWSRegion>", # The AWS region for the VPC
credentials_id="<CredentialsId>", # Your credential id
storage_configuration_id="<StorageConfigurationId>", # Your storage configuration id
network_id="<NetworkId>", # Your network id
workspace_name="<WorkSpaceName>",) # Name for your workspace
- how should I retrieve the AWS region?
-
https://www.pulumi.com/docs/concepts/config/
- should I us the require concept:
name = config.require('name');
?
- or can these be auto-resolved from a configured AWS CLI?
- how to set the following:
- credentials_id
- storage_configuration_id
- network_id
I think these might require setting up some more S3 buckets or VPCs for databricks? But am unsure what to put there.
Regarding the cluster: I think
https://docs.databricks.com/en/dev-tools/pulumi.html is already describing how to set up the cluster. But if you could include a mini dummy cluster example this would be very helpful as well.
