Breadcrumbs

Amazon S3 - Download Files to Platform

Overview​

Quickly export one or more files from an Amazon S3 bucket. Once the files have downloaded, transfer them to another service or run another Vessel against the data.

Variables​

Name

Reference

Type

Required

Default

Options

Description

Bucket Name

S3_BUCKET_NAME

Alphanumeric

✅

-

-

The target S3 bucket the target file is stored in.

S3 Folder Name

S3_SOURCE_FOLDER_NAME

Alphanumeric

➖

-

-

Name of the folder where the file is stored in the S3 Bucket. If left blank, looks in the root directory.

S3 File Name Match Type

S3_SOURCE_FILE_NAME_MATCH_TYPE

Select

✅

exact_match

Exact Match: exact_match

Regex Match: regex_match

Determines if the text in "S3 File Name" will look for one file with exact match, or multiple files using regex.

S3 File Name

S3_SOURCE_FILE_NAME

Alphanumeric

✅

-

-

Name of the target file in the S3 bucket. Can be regex if "Match Type" is set accordingly.

Workflow Folder Name

S3_DESTINATION_FOLDER_NAME

Alphanumeric

➖

-

-

Folder where the file(s) should be downloaded on Platform. Leaving blank will place the file in the home directory.

Workflow File Name

S3_DESTINATION_FILE_NAME

Alphanumeric

➖

-

-

What to name the file(s) being downloaded on Platform. If left blank, defaults to the original file name(s).

Access Key ID

AWS_ACCESS_KEY_ID

Password

✅

-

-

The access key ID for programmatic IAM user used to download the file. See Authorization documentation for more information.

Secret Access Key

AWS_SECRET_ACCESS_KEY

Password

✅

-

-

The secret access key for programmatic IAM user used to download the file. See Authorization documentation for more information.

Region

AWS_DEFAULT_REGION

Select

✅

us-east-2

us-east-2,us-east-1,us-west-1,us-west-2,af-south-1,ap-east-1,ap-south-1,ap-northeast-3,ap-northeast-2,ap-southeast-1,ap-southeast-2,ap-northeast-1,ca-central-1,cn-north-1,cn-northwest-1,eu-central-1,eu-west-1,eu-west-2,eu-south-1,eu-west-3,eu-north-1,sa-east-1,me-south-1,

The AWS region for the S3 bucket and IAM user.

YAML​

Below is the YAML template for this Template and can be used in the Fleet YAML Editor.

source:
  blueprint: Amazon S3 - Download Files to Shipyard
  inputs:
    S3_BUCKET_NAME: null  ## REQUIRED
    S3_SOURCE_FOLDER_NAME: null
    S3_SOURCE_FILE_NAME_MATCH_TYPE: exact_match ## REQUIRED
    S3_SOURCE_FILE_NAME: null ## REQUIRED
    S3_DESTINATION_FOLDER_NAME: null
    S3_DESTINATION_FILE_NAME: null
    AWS_ACCESS_KEY_ID: null  ## REQUIRED
    AWS_SECRET_ACCESS_KEY: null ## REQUIRED
    AWS_DEFAULT_REGION: us-east-2 ## REQUIRED
  type: BLUEPRINT
guardrails:
  retry_count: 1
  retry_wait: 0h0m0s
  runtime_cutoff: 1h0m0s
  exclude_exit_code_ranges:
    - '101'
    - '102'
    - '103'
    - '104'
    - '105'
    - '106'
    - '107'
    - '249'