← All examples

Storage

Share files between jobs on a volume

A volume is a persistent folder mounted into every container that asks for it. Stages of a pipeline hand files to each other through it, notebooks keep their work on it, and you browse, upload and download from the dashboard or the CLI.

JuiceFS over object storage (S3 on the lab cluster); a CloudBucket mounts an S3 or GCS bucket directly.

How it works

  1. 1Volume(name=, mount_path=) in a decorator mounts it; the same name from different functions is the same folder.
  2. 2Storage in the dashboard lists volumes with a file browser, previews, upload and download, and where each one is mounted.
  3. 3levers9 ls, cp, mv, rm work on VOLUME/path; cp uploads a folder in one go.
  4. 4Datasets, notebooks and pipeline artifacts all live on volumes, so any of them can be read by any function.

The code

demo/volumes/app.py
from levers9 import Volume, job

DATA = Volume(name="datasets", mount_path="/data")

@job(volumes=[DATA])
def count():
    import os
    return len(os.listdir("/data/uploads"))
run it
levers9 volume create datasets
levers9 cp ./recordings levers9://datasets/uploads
levers9 ls datasets/uploads

Where to go next

Related