LLM serving

Deploy an OpenAI-compatible LLM endpoint

Pick an open-weights model, pick a GPU, deploy. levers9 runs vLLM for you and gives you a URL that speaks the OpenAI API, so every client that takes a base_url works unchanged.

One GPU per replica: L40S for 12B-class models, A10G or L4 for 4B to 8B. Scales on token pressure, to zero when idle.

Open Endpoints LLM serving docs

How it works

1The Endpoints page lists open-weights models verified against the Hugging Face Hub and suggests a GPU from the parameter count.
2Deploy starts a vLLM container on the chosen GPU; the first start downloads the weights to a volume, later starts reuse them.
3The controlplane routes the endpoint's URL to the container and meters it as an OpenAI-compatible service.
4From Python the same thing is the VLLM integration; the CLI has levers9 deploy --llm.

The code

app.py

from levers9 import QueueDepthAutoscaler
from levers9.integrations import VLLM, VLLMArgs

llm = VLLM(
    name="gemma",
    gpu="L40S", cpu=4, memory="24Gi",
    vllm_args=VLLMArgs(model="google/gemma-4-12B-it", max_model_len=8192),
    secrets=["HF_TOKEN"],
    keep_warm_seconds=300,
    autoscaler=QueueDepthAutoscaler(max_containers=2),
)

run it

levers9 deploy app.py:llm --name gemma

# then, from any OpenAI client
client = OpenAI(base_url="https://levers9.uselevers.com/gemma/v1", api_key=TOKEN)

Where to go next

Open Endpointsthe catalog, one-click deploy LLM serving docsVLLM, VLLMArgs, the CLI flags Endpoints in the dashboardwhat the page does

Ship a model as an HTTP endpoint

One decorator turns a function into a URL with keep-warm and autoscaling; deploy from the CLI with hot reload while you build.

Sandboxed code execution

Run LLM-generated code in gVisor-isolated containers with CPU, memory and network limits per call.

Teach a call agent with human feedback

Play the seven missions: calls in, hot spots, duels for 10 XP a pick, level up, boss fight, crown the champion, connect it to real customers.