Pipelines

Train four models in parallel on L40S

A training pipeline in plain Python: prep the data on a CPU container, train four configurations at once with one GPU container each, evaluate them on the GPU, package the winner. No workflow engine, no YAML; .remote() and .map() are the whole orchestration.

CPU containers for prep and package, one GPU container per training config. Ran on L40S; runs on any GPU type in the cluster.

GPU pipeline docs Demo source

How it works

1prep.remote() runs once on CPU and writes the dataset to a volume shared by every stage.
2train.map(grid) starts one GPU container per config; they run in parallel when the pool has capacity and queue otherwise.
3evaluate.remote(models) scores every checkpoint on the same held-out set and returns a leaderboard.
4package.remote() bundles the best model as a release artifact on the volume, ready to download with levers9 cp.

The code

demo/gpu_pipeline/app.py

ART = Volume(name="gpu-pipeline-artifacts", mount_path="./artifacts")
TORCH = Blueprint(python_version="python3.11", python_packages=["torch"])
CPU = dict(cpu=1, memory="512Mi", volumes=[ART])
GPU = dict(gpu="L40S", gpu_count=1, cpu=2, memory="8Gi", image=TORCH, volumes=[ART])

@job(**CPU)
def prep(n_rows: int) -> dict: ...

@job(**GPU)
def train(config: dict) -> dict: ...

@job(**GPU)
def evaluate(models: list) -> dict: ...

def run_pipeline(version: str):
    data = prep.remote(n_rows=100_000)
    models = list(train.map([{"name": "small", "hidden": 64}, {"name": "wide", "hidden": 512}, ...]))
    best = evaluate.remote(models)["best"]
    return package.remote(best["name"], version)

run it

cd demo/gpu_pipeline
python app.py                       # the four stages, with each training's epochs streaming back
levers9 ls gpu-pipeline-artifacts   # the release tarball

Where to go next

GPU pipeline docsthe demo, step by step Demo sourcedemo/gpu_pipeline/app.py Appsevery stage as a task with its logs

Process ten thousand inputs with .map()

One container per item, sub-second warm starts, scale back to zero when the queue drains.

Run a pipeline every night

A cron expression on a function; the whole prep, train, evaluate chain runs unattended and lands on a volume.

Share files between jobs on a volume

Persistent folders mounted into every container; browse, upload and download from the dashboard or the CLI.