Serve Engine

Inference on your machines.

SMILE Serve is the inference runtime all you need. A commercial license covers production use without the AGPL network source offer.

Start Read the license

What you run

Classic models, ONNX graphs, and chat run in the same process, on your machines and your network. A React UI is served from that process.

Classic ML

Serialized SMILE models (.sml) — classifiers and regressors — behind /api/v1/smile.

ONNX Runtime

Any model in the ONNX open format (.onnx), including computer vision, behind /api/v1/onnx.

LLM Chat

OpenAI-compatible chat completions, tool calling, and conversation history behind /api/v1/chat.

Web UI

A bundled interface for chat, SMILE, and ONNX models, served from the same process.
Bash 
docker run -it \
  -v /path/to/model/folder:/model \
  -p 8888:8080 \
  ghcr.io/haifengl/smile-serve:latest

For agent loops

While Chatbots optimize for high throughput and long context streaming, agentic workflows care about something different: TTFT (Time to First Token) on frequent short prompts, rapid tool-call latency, and predictable memory overhead when running parallel loops. Our Serve Engine is built for that: a fast first token, a fast tool round trip, and a memory budget reserved when the model loads.

Tool results stay in the prefix

A tool result extends the same context. The cache is not rebuilt, and it does not grow one unbounded copy at a time.

Short turns are not held back

A request is admitted when its own prompt and token window fit. An interactive loop is not stalled until a large cohort fills.

Memory is reserved up front

KV and speculation checkpoints are sized at load. A long session does not grow the pool one request at a time.

The draft keeps its prefix

The draft head holds the prefix instead of recomputing it every round.

No second forward

On a partial accept, saved DeltaNet state is restored instead of another forward pass.

The gain holds in a full batch

Other serving engines lose the speculative advantage as the batch fills. SMILE holds it.

How it compares

vLLM

SMILE matches or slightly exceeds vLLM in raw decode throughput while outperforming it in speculative decoding scaling and hybrid architecture state management.

SGLang

SMILE achieves comparable token throughput and FlashInfer integration efficiency while offering a tighter execution pipeline for native speculative heads like MTP.

TensorRT-LLM

SMILE achieves TRT-LLM levels of low-level optimization (CUDA graphs for ragged verification batches, Marlin INT4 execution) without requiring heavy C++ compilation workflows or rigid engine builds.