App Process & Development Deep Dive

Note the following document expects an understanding of “Client Credentials”. Instructions for utilizing client credentials is not included in this document.

An App is the "unit of work" in Alli Marketplace

Most importantly, an App has a type. The default available app types are as follows:

• Redshift Cache - Redshift Cache Apps allow you to create caches of views in the Datawarehouse. A cache, in this case, is a copy of the entire view data into a separate table. Since tables are much faster to query, and views require calculations, this cache data is much faster to query. The downside is that the cached tables are not updated when the view is. So data can become stale. It is up to you to figure out the trade-off, and how often to schedule the App in order to have data that is 1. recent and 2. fast to query.

• Python - Python Apps allow you to run programs written in Python. Currently, only Python3 is supported. Any program you write in Python can be uploaded and run inside Marketplace. In the App configuration, there are options for Environment Variables and command-line arguments for your Python program. 

• Bash - Bash Apps allow you to run programs written in the Bash scripting language. Any program that you write in Bash can be uploaded and run inside Marketplace. In the App configuration, there are options for Environment Variables and command-line arguments for your Bash scripts. 

Currently, these are the available App types. Each type has its own purpose and configuration parameters. Please see the links from the above section for a deeper dive into each type and their respective usage and configuration. 

Getting Started

All Apps have a configuration. Part of this configuration is common to all Apps, independent of type. All Apps also have a subset of configuration that is dependent on the App's type. Here we examine the common configuration for all Apps.

• To create an App, navigate to the App list page and click the Add App button.

• On the following page, you are able to select the type of App you want. 

• The configuration on the left side of the Edit App page is common to all App types. 

Name - Name of the App. This shows up in listings and on Execution pages so you know which App you are working with. App names must be unique to a Client.

Status - The active or inactive status of the App. Apps that are not active are not scheduled/Executed even if it still has Scheduled. Active Apps have Executions that run according to each of their Schedules. When saving an App and the Status changes from active to inactive, all current Executions that have not started are deleted. 

Schedules - A list (row by row) of schedules that Executions for the App are run. The first column is the Weekday on which to run the Execution. The Daily option on a schedule means run every single day at the specified time. The second column is the time of day at which the Execution. 15-minute increments are available for the entirety of a day. 

Configuration for Individual Types

Environment Variables - Environment Variables are variables that are set across all running programs. They can be set or removed for individual programs. These types of variables are the preferred way to pass secrets (password, API tokens, etc.) to programs. 

Environment Variables are like variables in other programming languages. They all have a name and a value. By referencing the name in your program, you can retrieve the value. There are usually special manners with which to retrieve Environment Variables as opposed to "normal" variables in a programming language. See each App type's article for how to access them. 

Secrets required by your programs should never be hard-coded into the program. Instead, use Environment Variables to pass them to programs. Environment Variables set in Marketplace are not visible in the UI and are encrypted. 

In the above widget, each Environment Variable is set per row, with the provided Name and Value. When you first create an App, the Values should be filled out. You can toggle whether or not to view the input by clicking the eye icon. After an App has been created and you are editing it, the Values are going to appear empty. This is done for security reasons so that the secrets are not downloaded to your browser. By leaving the Value blank, Marketplace will not update the Value for a Name that already exists. 

See the Environment Variables section in the App Deep Dive Article for more information. 

File - A File is a file that you can upload to an App that contains the code and resources the App needs to do its job. Following is a screen capture of the File widget. 

The filename, here script.py, is the name of the file that you select from your device. The file name remains present until you save or choose to cancel a file upload. When creating an App with this field, it is required. When editing an existing App, the file name does not show up, but the file still exists in Marketplace and will be part of the App's configuration until you change it. 

Command - A Command is a list (array) of string values that are used to execute a program within an operating system. This list of arguments always starts with the name of the program to run. All remaining arguments are passed to the invoked program and are available in the program's code. In this manner, you can affect how the program works by giving it different arguments. 

Commands are provided into Marketplace as a comma-separated list of arguments in a single input. Because of the CSV format, spaces and quotations at the beginning and end of individual fields are important. 

E.g. the Command bash, ./script.sh, file.txt results in the following arguments, notice the space at the beginning of items 2 and 3:

1. bash

2. ./script.sh

3. file.txt

Whereas the Command bash,./script.sh,file.txt results in the following, different arguments:

1. bash

2. ./script.sh

3. file.txt

The leading spaces in the first example are going to be passed as is in an Execution. You will most likely want the second example where spaces are not present. Those leading spaces are not a part of the file name, arguments, etc. thus including them will likely cause errors. 

Command arguments are not suitable for secrets. They are available to change the behavior of your program with values that are not secret, like a file name to process or a column name in a table to work with. 

Saving an App

On the New and Edit App pages:

Clicking the Save & Run buttons will save the App and schedule an Execution to run right now. If the App is active and has schedules, then more Executions may be created and scheduled at their appropriate times. However, just one Execution will be scheduled for the current time. 

Clicking the Save button saves the App without scheduling an Execution for right now. If the App is active and has schedules, then more Executions may be created and scheduled at their appropriate times, but none for right now. 

Deep Dive

What is a Process? 

It should be noted that the documentation below describes the Linux operation system kernel. It may or may not apply to other operating systems. Linux is the operating system used in Marketplace. 

A Process (in a computer) is a program in execution. The program is a sequence of instructions (that the processor interprets and executes). And an instance of those instructions alongside all associated data (in execution) is a process. 

Obviously, there is a lot more to examine in a process that is outside of the scope needed to write things for Marketplace, but, if you need to remember one thing about a process it is this: "A process is a program (list of instructions) in execution."

The Life-cycle of a Process

A process is started when another (parent) process chooses to do so. The first process (init) in Linux systems starts all other processes and is an ancestor to every program on your computer. 

Let's say you are in a bash terminal, and you type `date` and then hit Enter, the following happens:

1. Bash reads the input and realizes it should execute the date program for you. 

2. It talks to the operating system requesting a date process be started. 

3. Bash waits for the date process to finish. It relays all input and output to the terminal you are in so that you can see what the date process is doing. 

Bash is called a shell because it is the outermost layer around the core operating system that the user interfaces with to use the computer. 

Process Attributes

Exit Codes

From above, we can see that a Process, in general, is started, does some executing, then exits. 

This brings up the first attribute - the exit code. Every process has an exit code that determines the success or failure of the process. It is up to the program (it's code) to determine whether or not its execution was successful. 

The available exit codes are 0 - 255 inclusive. An exit code of 0 means successful execution whereas non-zero codes denote "failure." Programs can use different exit codes to denote different types of failure. 

For instance, a copy program that copies a file from one file-system location to another could fail with exit code 1 if there was an issue with the destination file, and exit code 2 if there was an issue with the source file. Other programs that know how copy works can examine the exit code to know what went wrong. 

Exit Codes in Marketplace

The exit codes in Marketplace are used to determine the success or failure of an Execution. An Execution, along with its meta-data in Marketplace, boils down to a process that was executed in some Linux operation system. 

For instance, if you have a Python App in Marketplace, the exit status of your Python program becomes the exit status of the Execution and thus determines success. Remember, an exit code of 0 means everything was successful. 

As another example, the Redshift Cache App type does not allow you to write and execute a program - it does that for you. But, the program that runs for you uses the exit code to tell Marketplace whether or not the caching was completed correctly. 

Standard Output and Standard Error

Every process has some files associated with it that you can and should use in Marketplace that will show up on an Execution's page. 

Standard Output has a file descriptor of 1. Depending on the language you are using, the most simple way to print data to output is going to be to this file. The programming language handles this for you. 

For example, in Python3, the line print('Hello, Marketplace') would print that string to standard output. 

Standard Error has a file descriptor of 2. The manner in which you print to this file (which still shows up and looks the same as standard output) is usually more complicated than standard output and depends on the programming language. Refer to your specific programming language's documentation if you need to do this. 

Standard Output and Standard Error in Marketplace

The combined result of standard output and standard error show up as Output on the view Execution page. 

Working Directories

Processes have something called a Working Directory. This is a directory (folder) on the file system that is associated with the process by the operating system. Note that there are not any artifacts in the file system for a process' working directory. 

When files with a relative path are used, then references are calculated from the working directory. For instance, if you are in the /home/Marketplace/foo directory and open a file bar.txt the full path will be /home/Marketplace/foo/bar.txt.

You can change the working directory of a process by using your programming language's libraries. 

Working Directories in Marketplace

When your code runs in Marketplace, the working directory of the process for your code is /home/forklift/<exec_id> where <exec_id> is the running Execution's Id. 

If applicable, this is where downloaded files are located. 

Command Arguments

Command arguments are values that can be used to alter the functionality of your program. When a process is started, it is done so with an array of string arguments that programs can get and parse from language libraries. Refer to the documentation of your specific language to understand how to use command arguments. 

As an example, the ls command prints files in a directory. To alter the output per file you can add a -l command argument. To list all (hidden) files, you can add -a to the arguments. Thus running ls -l -a provides different functionality than simply running ls. The ls program knows how to interpret those arguments and what they mean to alter its output. 

Command Arguments in Marketplace

You can and should use command arguments to alter the behavior of your program if those arguments are non-secret. If applicable, the command arguments can be listed on the Edit App page for Apps that accept that configuration. 

Environment Variables

Environment variables are key, value pairs available to a process. They are set outside of the process or program. 

As with all variables, the point is to carry values via a name so that programs are able to reference the values when need be. The usual use case for environment variables is that of passing secrets to a program. Because command arguments (i.e. --password=qwerty1234?) are available to other programs, can be found in special files, etc. it is not recommended to pass secrets via command arguments. 

Environment variables, however, are not accessible in that manner and are much better suited for passing secrets to a program. For instance, if you have a secret API key for some service, then setting the environment variable and having the program access it via libraries is preferred. 

You will need to refer to the documentation for your specific programming language to figure out how to obtain those values. 

Environment Variables in Marketplace

If applicable, depending on the App type, there are environment variables that you can define to be set for your program. These values should be used for secrets and as minimally as possible. To alter the functionality of your program in a non-secret manner (e.g. number of iterations, keyword to search for) you should use command arguments. 

In addition to the environment variables that you can set on your Apps, there are some environment variables that Marketplace sets for you in order to for you to determine what execution, App, client, etc. you are running for. 

• FORKLIFT_CLIENT_ID - Set to the id of the Client that the App / Execution belongs to. 

• FORKLIFT_CLIENT_NAME - Set to the name of the Client that the App / Execution belongs to. 

• FORKLIFT_App_ID - Set to the id of the App that the Execution belongs to.

• FORKLIFT_App_NAME - Set to the name of the App that the Execution belongs to. 

• FORKLIFT_EXEC_ID - Set to the id of the Execution that is running. 

• FORKLIFT_EXEC_WORKDIR - Set to the working directory that your App is set to do its work in.