Databricks SQL Warehouse - Download Query Results to Workflows
Overview
Quickly download the results of Databricks SQL query to Workflows. You can choose to output the results as a CSV or a Parquet file. __Recommended Setup__ Although they are not required in order to connect, it is recommended that you provide the `Catalog` and the `Schema` that you will query. By not doing so, the connection will resort to the defaults and it is possible that the query will fail if the tables don't reside in the default schema and catalog. **NOTE** This will not download the file to your local computer, but to the home directory within Workflows
Variables
Name | Reference | Type | Required | Default | Options | Description |
---|---|---|---|---|---|---|
Access Token | DATABRICKS_SQL_ACCESS_TOKEN | Password | ✅ | - | - | The access token generated in Databricks for programatic access |
Databricks Server Host | DATABRICKS_SQL_SERVER_HOST | Alphanumeric | ✅ | - | - | The URL address of the SQL warehouse |
Warehouse HTTP Path | DATABRICKS_SQL_HTTP_PATH | Alphanumeric | ✅ | - | - | The extended path for the SQL warehouse |
Catalog | DATABRICKS_SQL_CATALOG | Alphanumeric | ➖ | - | - | The optional catalog to connect to. If none is provided, this will default to Hive Metastore |
Schema | DATABRICKS_SQL_SCHEMA | Alphanumeric | ➖ | - | - | The optional schema to connect to. If none is provided, the blueprint will connect to the `default` schema |
Query | DATABRICKS_SQL_QUERY | Alphanumeric | ✅ | - | - | The query to send to Databricks |
Workflows Folder Name | DATABRICKS_SQL_FOLDER_NAME | Alphanumeric | ➖ | - | - | The optional name of the folder where the file in Workflows is located |
Workflows File Name | DATABRICKS_SQL_FILE_NAME | Alphanumeric | ✅ | - | - | The name of the outputted file |
File Type | DATABRICKS_SQL_FILE_TYPE | Select | ✅ | csv | CSV: csv Parquet: parquet | The file type to load |
YAML
Below is the YAML template
source:
template: Databricks SQL Warehouse - Download Query Results to Workflows
inputs:
DATABRICKS_SQL_ACCESS_TOKEN:
DATABRICKS_SQL_SERVER_HOST:
DATABRICKS_SQL_HTTP_PATH:
DATABRICKS_SQL_CATALOG:
DATABRICKS_SQL_SCHEMA:
DATABRICKS_SQL_QUERY:
DATABRICKS_SQL_FOLDER_NAME:
DATABRICKS_SQL_FILE_NAME:
DATABRICKS_SQL_FILE_TYPE: csv
type: TEMPLATE
guardrails:
retry_count: 1
retry_wait: 0h0m0s
runtime_cutoff: 1h0m0s
exclude_exit_code_ranges:
- 200
- 202
- 203
- 204
- 205
- 206
- 207
- 208
- 209
- 210
- 211
- 249