Databricks - Upload Files to DBFS from Workflows

Overview

Easily import one or more files from Workflows to your Databricks File System (DBFS). The [match type](https://www.shipyardapp.com/docs/reference/blueprint-library/match-type/) selected greatly affects how this Blueprint works.

Variables

Name	Reference	Type	Required	Default	Options	Description
Workflows Folder Name	DATABRICKS_SOURCE_FOLDER_NAME	Alphanumeric	➖	None	-	The local folder name where the file is contained
Workflows File Name Match Type	DATABRICKS_SOURCE_FILE_NAME_MATCH_TYPE	Select	✅	exact_match	Exact Match: exact_match Regex Match: regex_match	Determines if the text in "Workflows File Name" will look for one file with exact match, or multiple files using regex.
Workflows File Name	DATABRICKS_SOURCE_FILE_NAME	Alphanumeric	✅	None	-	Name of the target file on Workflows. Can be regex if "Match Type" is set accordingly.
Databricks Folder Name	DATABRICKS_DESTINATION_FOLDER_NAME	Alphanumeric	➖	None	-	Name of the folder where you want to upload the local file(s) in the Databricks File System (DBFS). If left blank, uploads to /FileStore/.
Databricks File Name	DATABRICKS_DESTINATION_FILE_NAME	Alphanumeric	➖	None	-	What to name the file(s) being uploaded. If left blank, defaults to the original file name(s).
Workspace Instance URL	DATABRICKS_INSTANCE_URL	Alphanumeric	✅	None	-	The subdomain, domain, and top-level domain (TLD) of your Databricks Workspace URL.
Access Token	DATABRICKS_ACCESS_TOKEN	Password	✅	None	-	The personal access token associated with the provided Workspace Instance.

YAML

Below is the YAML template

YAML


source:
  template: Databricks - Upload Files to DBFS from Workflows
  inputs:
    DATABRICKS_SOURCE_FOLDER_NAME:
    DATABRICKS_SOURCE_FILE_NAME_MATCH_TYPE: exact_match
    DATABRICKS_SOURCE_FILE_NAME:
    DATABRICKS_DESTINATION_FOLDER_NAME:
    DATABRICKS_DESTINATION_FILE_NAME:
    DATABRICKS_INSTANCE_URL:
    DATABRICKS_ACCESS_TOKEN:
  type: TEMPLATE
guardrails:
  retry_count: 1
  retry_wait: 0h0m0s
  runtime_cutoff: 1h0m0s
  exclude_exit_code_ranges:
    - 200
    - 201
    - 202
    - 203
    - 212
    - 214