Google BigQuery - Download Query Results to Workflows

Overview

Larger datasets may run into file size limitations set by BigQuery. In these instances, we recommend using the Google BigQuery - Store Query Results in Google Cloud Storage Template.

Variables

Name	Reference	Type	Required	Default	Options	Description
Query	BIGQUERY_QUERY	Alphanumeric	✅	-	-	Standard SQL query to be executed against BigQuery. Does not support Legacy SQL.
Shipyard File Name	BIGQUERY_DESTINATION_FILE_NAME	Alphanumeric	✅	-	-	Name of file to be generated with the results. Should be `.csv` extension.
Shipyard Folder Name	BIGQUERY_DESTINATION_FOLDER_NAME	Alphanumeric	➖	-	-	Folder where the file should be downloaded. Leaving blank will place the file in the home directory.
Service Account	GOOGLE_APPLICATION_CREDENTIALS	Password	✅	-	-	JSON from a Google Cloud Service account key.

YAML

Below is the YAML template for this Template and can be used in the Fleet YAML Editor.

CODE

source:
  blueprint: Google BigQuery - Download Query Results to Shipyard
  inputs:
    BIGQUERY_QUERY: null  ## REQUIRED
    BIGQUERY_DESTINATION_FILE_NAME: null ## REQUIRED
    BIGQUERY_DESTINATION_FOLDER_NAME: null
    GOOGLE_APPLICATION_CREDENTIALS: null ## REQUIRED
  type: BLUEPRINT
guardrails:
  retry_count: 1
  retry_wait: 0h0m0s
  runtime_cutoff: 1h0m0s
  exclude_exit_code_ranges:
    - '101'
    - '103'
    - '104'
    - '200'
    - '203'
    - '205'

Overview​

Variables​

YAML​

Overview

Variables

YAML