Storage
Share files between jobs on a volume
A volume is a persistent folder mounted into every container that asks for it. Stages of a pipeline hand files to each other through it, notebooks keep their work on it, and you browse, upload and download from the dashboard or the CLI.
JuiceFS over object storage (S3 on the lab cluster); a CloudBucket mounts an S3 or GCS bucket directly.
How it works
- 1Volume(name=, mount_path=) in a decorator mounts it; the same name from different functions is the same folder.
- 2Storage in the dashboard lists volumes with a file browser, previews, upload and download, and where each one is mounted.
- 3levers9 ls, cp, mv, rm work on VOLUME/path; cp uploads a folder in one go.
- 4Datasets, notebooks and pipeline artifacts all live on volumes, so any of them can be read by any function.
The code
demo/volumes/app.py
from levers9 import Volume, job DATA = Volume(name="datasets", mount_path="/data") @job(volumes=[DATA]) def count(): import os return len(os.listdir("/data/uploads"))
run it
levers9 volume create datasets levers9 cp ./recordings levers9://datasets/uploads levers9 ls datasets/uploads
Where to go next
Related
Train four models in parallel on L40S
Prep on CPU, fan out training to one GPU container per config, evaluate, package: a pipeline in plain Python.
Transcribe calls in batches with Whisper
Drop recordings on a volume, get diarised transcripts as a dataset, with a person correcting the drafts.
Open a JupyterLab notebook on a GPU
Your volumes and secrets mounted, live CPU, memory and GPU meters, the kernel on the cluster.