How to Read and Write Files to a Client's S3 Bucket

Like connecting to redshift, uploading files to S3 is done via environment variables:

ALLI_S3_BUCKET
ALLI_S3_KEY_PREFIX
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

Alli Marketplace provides AWS Credentials that are scoped to a Client via AWS_* environment varaibles listed above. These will “just work” ™ with AWS SDKs that look for credentials in the environment by default.

Uploading Files

To upload a file to s3, create a s3 client object and use the ALLI_S3_BUCKET and ALLI_S3_KEY_PREFIX in the request.

Failure to use ALLI_S3_KEY_PREFIX will result in your upload failing. Alli Marketplace can read from anywhere in the bucket, but may only write to certain locations.

PY

import os
import boto3

s3 = boto3.client('s3')

s3.put_object(
  Bucket=os.environ['ALLI_S3_BUCKET'],
  Key=os.environ['ALLI_S3_KEY_PREFIX']+'your_filename_here.csv',
  # These are required when use Alli Client Credentials
  ACL='bucket-owner-full-control',
)

Note that no leading slashes are required your filename: concatenate the ALLI_S3_KEY_PREFIX with the custom filename and it’s good to go.

Uploading Temporary Files

Use the ALLI_S3_KEY_TEMP_PREFIX environment variable:

PY

import os
import boto3

s3 = boto3.client('s3')

s3.put_object(
  Bucket=os.environ['ALLI_S3_BUCKET'],
  Key=os.environ['ALLI_S3_KEY_TEMP_PREFIX']+'your_filename_here.csv',
  # These are required when use Alli Client Credentials
  ACL='bucket-owner-full-control',
)

These files will be deleted after 7 days.

Download Files

This works the same way as uploading files to the Client’s S3 Bucket, but no need to worry about key prefix. Alli Marketplace can read from anywhere in the client’s bucket.

PY

import os
import boto3

s3 = boto3.client('s3')

object = s3.get_object(Bucket=os.environ['ALLI_S3_BUCKET'], Key='some_key.csv')