Within Marketplace, find the Redshift to BigQuery application in the browse section. Install the application and when setting up this app you will need to configure various fields. These are documented below and within the app details:
REDSHIFT_OBJshould be used, if both are used it will error out.
This App is best used in the following scenarios:
At the end of an analytical workflow where you need to move the final result to BigQuery
One Off Data Load requests.
PARTITON_COLSHOULD be used for all Data Copies that include a date column
This will allow BigQuery to trim unneeded partitions during query time
The SQL that generates the result that needs to be migrated to BigQuery
select * from gap_core.search_account where date >='2019-01-01'
The Table or View that needs to be migrated to BigQuery
The BigQuery Project to create the new table in
Y, Defaults to pmg-datawarehouse
The BigQuery Dataset that the new table needs to be created in
The BigQuery Table Name
The Column that should partition the data
N, SHOULD be used in most instances
The GOOGLE_APPLICATION_CREDENTIALS FIle.
Y, Defaults to pmg-datawarehouse creds
The Target Schema File.
Comma delimited list of columns to cluster by. For Best Practices on when to cluster see the BigQuery Documentation Here: https://cloud.google.com/bigquery/docs/clustered-tables
Drops Target Table before attempting to load it.
Y, Defaults to FALSE
The Redshift Host you need to connect to
The Redshift Database
The Redshift User that has access to the data
If you provide a value for REDSHIFT_HOSTNAME, the other Redshift Template Variables will need to be populated, as well as you’ll need to add an Environment Variable named
REDSHIFT_PASSWORD with the password for the corresponding
Features Change log
Added the ability to pull data from any Redshift Cluster
Added Cluster by Column(s) Option to help speed up BigQuery Query Performance
Added Drop Table Option
Auto-Generate BigQuery Schema (Only if no schema is provided by the user)
Table Mapping of Pandas Dataframe dtype to BigQuery Data Type:
BigQuery Data Type
Data will always load into BigQuery as Truncate/Insert
Data being migrated is less than 7 million rows