LLM serving

Ship a model as an HTTP endpoint

One decorator turns a function into an HTTP URL with keep-warm and autoscaling. While you build, levers9 serve gives you a temporary URL with hot reload; levers9 deploy makes it a versioned URL that stays.

CPU or GPU; on_start loads a model once per container so requests do not pay for it.

Endpoint docs Parameters

How it works

1@endpoint wraps a function that takes the request body as keyword arguments and returns JSON.
2on_start runs once per container (load weights, open a connection); its return value reaches the handler as context.on_start_value.
3keep_warm_seconds keeps containers alive between requests; the autoscaler adds containers on queue depth.
4@asgi wraps a whole FastAPI or Starlette app the same way; @realtime handles WebSockets.

The code

demo/endpoint/app.py

from levers9 import endpoint, Blueprint

def load():
    from transformers import pipeline
    return pipeline("sentiment-analysis")

@endpoint(cpu=2, memory="4Gi", image=Blueprint(python_packages=["transformers", "torch"]),
          on_start=load, keep_warm_seconds=300)
def classify(context, text: str) -> dict:
    return context.on_start_value(text)[0]

run it

levers9 serve app.py:classify              # temporary URL, hot reload
levers9 deploy app.py:classify --name classify
curl -X POST .../endpoint/classify/v1 -d '{"text": "great service"}'

Where to go next

Endpoint docsserve, deploy, on_start, autoscaling Parameterskeep_warm, workers, autoscaler

Deploy an OpenAI-compatible LLM endpoint

Serve Qwen, Gemma or Llama with vLLM on an L40S, behind a drop-in replacement for the OpenAI API.

Background jobs with retries

A queue you feed from Python or a webhook; each task gets its own container and a retry policy.

Process ten thousand inputs with .map()

One container per item, sub-second warm starts, scale back to zero when the queue drains.