LM Lite

K.1

Overview

LM Lite runs multi-model inference on shared GPU infrastructure. It handles batch processing, replica management, and health-check orchestration, and it ships inside Conduit as the default runtime.

The point of it: several models share a single GPU with 1 to 2GB of runtime overhead, where vLLM carries roughly 15GB per instance. For small-to-medium models, that difference decides what fits on the card.

K.2

Batch processing

Concurrent requests collect into batches. A batch executes when it fills or when its timeout expires, whichever comes first.

ModelConfig(
    model_id,
    max_model_concurrency=50,             # max requests per batch
    model_batch_execute_timeout_ms=1000,  # execute batch after timeout
)

K.3

Replicas and readiness

Load distributes across replicas by round-robin scheduling.

LMLiteBlock(
    models=[...],
    replicas=2,  # requests distributed across replicas
)

Traffic only routes to healthy nodes. A block reports ready once its container health check passes.

while not block.ready:
    time.sleep(5)
# healthcheck passed, safe to send traffic

K.4

Memory

vLLM carries about 15GB of base overhead regardless of model size. For small models that is 7 to 8x more memory than the model needs.

The overhead budget is what lets several models share one card. vLLM needs a separate instance per model; LM Lite shares the infrastructure.

GPU memory          24GB
Model A (7B)       ~14GB
Model B (1B)        ~2GB
Model C (1B)        ~2GB
LM Lite overhead    ~2GB
-------------------------
Total              ~20GB  fits

K.5

Configuration

from conduit import ModelConfig
from conduit.conduit_types import GPUS
from conduit.runtime import LMLiteBlock

block = LMLiteBlock(
    models=[
        ModelConfig(
            "Qwen/Qwen3-4B",
            max_model_len=1400,
            max_model_concurrency=50,
            model_batch_execute_timeout_ms=1000,
        ),
        ModelConfig(
            "microsoft/phi-3-mini",
            max_model_len=2048,
            max_model_concurrency=25,
            model_batch_execute_timeout_ms=500,
        ),
    ],
    gpu=GPUS.NVIDIA_L4,
    replicas=2,
)

K.6

Lifecycle

# initialize
block = LMLiteBlock(models=[...], replicas=2)

# clean stale instances
LMLiteBlock.gc()

# wait for readiness
while not block.ready:
    time.sleep(5)

# execute inference
result = block(model_id="Qwen/Qwen3-4B", input=..., output=...)

# cleanup
block.delete()

K.7

GPU support

NVIDIA CUDA GPUs today. AMD ROCm support is planned.

from conduit.conduit_types import GPUS

GPUS.NVIDIA_L4
GPUS.NVIDIA_A10
GPUS.NVIDIA_A100
# ... additional types

K.8

Specifications