Skip to main content

Serverless runtime environment

With a Dagster+ Serverless deployment, you can customize the runtime environment where your code executes. You may want to:

Dagster uses PEX to package your code and deploy them on Docker images. You also have the option to disable PEX-based deploys and deploy using a Docker image instead of PEX.

Use a different Python version​

The default Python version for Dagster+ Serverless is Python 3.8. Python versions 3.9 through 3.12 are also supported. You can specify the Python version you want to use in your GitHub or GitLab workflow, or by using the dagster-cloud CLI.

In your .github/workflows/deploy.yml file, update the PYTHON_VERSION environment variable with your desired Python version:

Updating the Python version in deploy.yml
Loading...

Use a different base image​

When possible, you should add dependencies by including the corresponding Python libraries in your Dagster project's setup.py file:

Example setup.py
Loading...

When adding dependencies with the setup.py file isn't possible, you can build a custom base image:

note

Setting a custom base image isn't supported for GitLab CI/CD workflows out of the box, but you can write a custom GitLab CI/CD yaml file that implements the manual steps noted.

  1. Include dagster-cloud[serverless] as a dependency in your Docker image by adding the following line to your Dockerfile:

    RUN pip install "dagster-cloud[serverless]"

  2. Build your Docker image, using your usual Docker toolchain.

  3. Upload your Docker image to Dagster+ using the upload-base-image command. This command will print out the tag used in Dagster+ to identify your image:

    $ dagster-cloud serverless upload-base-image local-image:tag

    ...
    To use the uploaded image run: dagster-cloud serverless deploy-python-executable ... --base-image-tag=sha256_518ad2f92b078c63c60e89f0310f13f19d3a1c7ea9e1976d67d59fcb7040d0d6

  4. Specify this base image tag in you GitHub workflow, or using the dagster-cloud CLI:

    In your .github/workflows/deploy.yml file, add the SERVERLESS_BASE_IMAGE_TAG environment variable and set it to the tag printed out in the previous step:

    Setting a custom base image in deploy.yml
    Loading...

Include data files​

To add data files to your deployment, use the Data Files Support built into Python's setup.py. This requires adding a package_data or include_package_data keyword in the call to setup() in setup.py. For example, given this directory structure:

- setup.py
- quickstart_etl/
  - __init__.py
  - definitions.py
  - data/
    - file1.txt
    - file2.csv

If you want to include the data folder, modify your setup.py to add the package_data line:

Loading data files in setup.py
Loading...

Disable PEX deploys​

Prior to using PEX files, Dagster+ deployed code using Docker images. This feature is still available.

You can disable PEX in your GitHub or GitLab workflow, or by using the dagster-cloud CLI.

In your .github/workflows/deploy.yml file, update the ENABLE_FAST_DEPLOYS environment variable to false:

Disable PEX deploys in deploy.yml
Loading...

You can customize the Docker image using lifecycle hooks or by customizing the base image:

This method is the easiest to set up, and doesn't require setting up any additional infrastructure.

In the root of your repo, you can provide two optional shell scripts: dagster_cloud_pre_install.sh and dagster_cloud_post_install.sh. These will run before and after Python dependencies are installed. They're useful for installing any non-Python dependencies or otherwise configuring your environment.