Optimal Databricks Cluster Type for Daily Batch Processing

A

Azure Databricks has two types of clusters: interactive and automated. You use interactive clusters to analyze data collaboratively with interactive notebooks. You use automated clusters to run fast and robust automated jobs.

Example: Scheduled batch workloads (data engineers running ETL jobs)

This scenario involves running batch job JARs and notebooks on a regular cadence through the Databricks platform.

The suggested best practice is to launch a new cluster for each run of critical jobs. This helps avoid any issues (failures, missing SLA, and so on) due to an existing workload (noisy neighbor) on a shared cluster.

https://docs.databricks.com/administration-guide/cloud-configurations/aws/cmbp.html#scenario-3-scheduled-batch-workloads-data-engineers-running-etl-jobs

When it comes to batch processing in Azure Databricks, the type of cluster you use can have an impact on the performance and efficiency of the process. In general, there are three main types of clusters in Azure Databricks: automated, interactive, and high concurrency.

Automated clusters are designed to automatically spin up and down resources based on the workload. These types of clusters are typically used for ad-hoc data analysis or machine learning experimentation, where users need quick access to resources and don't want to worry about managing the underlying infrastructure. However, because these clusters are designed to be ephemeral, they may not be the best choice for batch processing jobs that run on a regular schedule.

Interactive clusters, on the other hand, are designed for interactive data exploration and development. These clusters are intended for users who need to run multiple queries or jobs against a data source, and want to have a persistent environment in which to work. While interactive clusters can be used for batch processing, they may not be the most efficient option since they are optimized for interactive use.

Finally, high concurrency clusters are designed to support large-scale data processing and machine learning workloads that require high levels of concurrency. These clusters are typically used in production environments where multiple users need to run jobs concurrently. While high concurrency clusters may be a good choice for batch processing, they may also be overkill if you only need to run a single job once a day.

In this case, since the job only needs to run once a day, an automated or interactive cluster could be used, depending on the specific requirements of the job. If the job requires a persistent environment in which to work, an interactive cluster may be a better choice. If the job only needs to run once a day and can be completed quickly, an automated cluster may be a more efficient option.