When defining a databricks Cluster in Pulumi how can I preve Pulumi Community #aws

When defining a databricks.Cluster in Pulumi how c...

sparse-optician-70334

09/22/2023, 8:51 AM

When defining a databricks.Cluster in Pulumi how can I prevent it from starting the cluster? Pulumi AI suggest

num_workers=0

but at least for single node personal clusters this does not work and Pulumi is starting the cluster. As a result executing pulumi takes a very long time. How can I simply only define the cluster - but not start it?

mysterious-raincoat-41852

09/22/2023, 6:47 PM

I think num_workers=0 is part of default configuration for single node https://docs.databricks.com/en/clusters/single-node.html. Also its Databricks API call so I think its not Pulumi starting the cluster but Databricks so I would check their docs. https://docs.databricks.com/api/workspace/clusters/create Try to specify autoscale, min=max_workers=0?

sparse-optician-70334

09/26/2023, 7:38 AM

no it is always spun up

mysterious-raincoat-41852

09/26/2023, 1:03 PM

I don’t see any API to “start” cluster (only restart though) so I think its behavior by design. Asked internally but not sure can get answer quickly

mysterious-raincoat-41852

09/26/2023, 9:00 PM

This is the suggestion (but it would not solve the waiting issue I think?) What will work is to give two consequent API calls: One for creating a new cluster and then one for terminating it right afterwards. POST /api/2.0/clusters/create POST /api/2.0/clusters/delete This will stop the cluster creation

sparse-optician-70334

09/27/2023, 10:35 AM

thanks

Open in Slack

Previous Next