Google BigQuery - Download Query Results to Workflows

Overview

Larger datasets may run into file size limitations set by BigQuery. In these instances, we recommend using the Google BigQuery - Store Query Results in Google Cloud Storage Template.

Variables

Name	Reference	Type	Required	Default	Options	Description
Query	BIGQUERY_QUERY	Alphanumeric	✅	None	-	Standard SQL query to be executed against BigQuery. Does not support Legacy SQL.
Workflows File Name	BIGQUERY_DESTINATION_FILE_NAME	Alphanumeric	✅	None	-	Name of file to be generated with the results. Should be `.csv` extension.
Workflows Folder Name	BIGQUERY_DESTINATION_FOLDER_NAME	Alphanumeric	➖	None	-	Folder where the file should be downloaded. Leaving blank will place the file in the home directory.
Service Account	GOOGLE_APPLICATION_CREDENTIALS	Password	✅	None	-	Either JSON from a Google Cloud Service account key or fill in with ${ALLI_GOOGLE_CLOUD_CREDENTIALS}, a built in workflow environment variable that will populate this input with valid credentials.
Location	BIGQUERY_LOCATION	Alphanumeric	➖	-	-	This is the location/region where your project is stored, which can be found in the setting section of "Mange Client". If unsure of exact location/region, fill in with ${ALLI_GOOGLE_CLOUD_BIGQUERY_LOCATION} and it will pull the value from client settings.
Project ID	BIGQUERY_PROJECT	Alphanumeric	➖	-	-	This is the project name used for big query connections, which can be found in the setting section of "Mange Client". If unsure of exact project name, fill in with ${ALLI_GOOGLE_CLOUD_PROJECT} and it will pull the value from client settings.

YAML

Below is the YAML template

YAML


source:
  template: Google BigQuery - Download Query Results to Workflows
  inputs:
    BIGQUERY_QUERY:
    BIGQUERY_DESTINATION_FILE_NAME:
    BIGQUERY_DESTINATION_FOLDER_NAME:
    GOOGLE_APPLICATION_CREDENTIALS:
    BIGQUERY_LOCATION:
    BIGQUERY_PROJECT:
  type: TEMPLATE
guardrails:
  retry_count: 1
  retry_wait: 0h0m0s
  runtime_cutoff: 1h0m0s
  exclude_exit_code_ranges:
    - 101
    - 103
    - 104
    - 200
    - 203
    - 205