Serve

Inference on your machines.

SMILE Serve is the inference runtime all you need. A commercial license covers production use without the AGPL network source offer.

Overview

SMILE Serve is a production-ready inference server built on Quarkus that brings together three complementary inference capabilities on the JVM. A React-based web UI is bundled and served from the same process.

Quick Start with Docker

The fastest way to run SMILE Serve is via the pre-built Docker image. Mount a local directory containing your model files and map the port:

docker run -it \
  -v /path/to/model/folder:/model \
  -p 8888:8080 \
  ghcr.io/haifengl/smile-serve:latest

The service starts on port 8080 inside the container (mapped to 8888 on the host). Place your .sml and .onnx model files in /path/to/model/folder; they are discovered automatically at startup.

GPU Build & Run

Build the JVM+GPU image from source and run it with CUDA graph decode enabled. To enable Google login, add -Dsmile.auth.google.client-id=... -Dsmile.auth.google.client-secret=... -Dsmile.auth.google.redirect-uri=... to JAVA_OPTS_APPEND yourself (see Authentication); keep those values out of shell history / docker inspect by sourcing them from environment variables rather than inlining literals.

./gradlew :serve:build
sudo docker build -f serve/src/main/docker/Dockerfile.jvm-gpu -t quarkus/smile-serve-gpu .
sudo docker run -i --rm --gpus all -p 8888:8080 \
  -e JAVA_OPTS_APPEND="-Dsmile.chat.model=Qwen/Qwen3.8-27B" \
  -u root -v "/llm/model":/model quarkus/smile-serve-gpu

Debug variant (adds CUDA_LAUNCH_BLOCKING=1 for synchronous CUDA error reporting; slower, use only when diagnosing a CUDA graph issue):

sudo docker run -i --rm --gpus all -p 8888:8080 \
  -e CUDA_LAUNCH_BLOCKING=1 \
  -e JAVA_OPTS_APPEND="-Dsmile.chat.model=Qwen/Qwen3.8-27B" \
  -u root -v "/llm/model":/model quarkus/smile-serve-gpu

LLM Decode Benchmarks

Steady-state continuous-batching decode on NVIDIA A100 with FlashInfer, smile.chat.max-batch-size=48, max_tokens: 512, greedy sampling (temperature: 0), and a uniform decode cohort (smile.chat.admit-coalesce-ms=50 for parallel load tests). Step is engine decode-step throughput (batch / step_ms); aggregate is AggregateTokenThroughput over a ~3 s window (includes streaming / logging overhead).

Llama 3.1 8B Instruct

Both runs used a full batch of 48 active requests at locked positions (e.g. positions=[417..417]). AWQ is ~28% faster at the same concurrency; lighter weights also leave more free KV slots for longer context.

Qwen3.8 27B

Hybrid Qwen3.5 stack (Gated DeltaNet + full attention) at the same concurrency and locked positions (e.g. positions=[466..466]). Absolute throughput is lower than 8B Llama as expected for a ~27B hybrid model across four GPUs; per-request generation stays ~17 tok/s with the batch saturated. Qwen AWQ via compressed-tensors / Marlin is not supported yet (detector currently treats cards such as cyankiwi/Qwen3.8-27B-AWQ-INT4 as dense).

Qwen3.8 27B with MTP speculative decoding

Qwen3.5/3.8 checkpoints ship a native multi-token-prediction (MTP) head. With smile.chat.speculative=true the engine drafts speculative-tokens tokens with that head and verifies them in one target forward pass; accepted tokens are byte-for-byte what greedy decoding would have produced (the output is lossless, up to rare bf16 near-tie flips between a 3-token window and single-token decode). MTP is a latency feature for low concurrency: it speeds up a lone request about 2-3x, and it automatically stays out of the way once several requests are active.

Setup: BF16, TP=4 on A100 40 GB, greedy, speculative-tokens=2 (the default depth of 3 measures the same or slightly better), prompt "What is the history of China?", 512 new tokens, warm run, wall time including prefill.

Before the draft head received its prefix context (it only saw the current round's own draft tokens), MTP was ~15 % slower than plain decode with a ~50 % accept rate (27.5 vs 32.5 tok/s in an earlier measurement on a quieter system; absolute numbers differ between sessions, compare within a table).

A round (draft + one verify + bookkeeping) takes ~48 ms and commits 2.5 tokens on average (mean accepted draft depth 1.5 of 2). Throughput gains hold at long context (6.6k-token prompt, 256 new tokens: 16.7 s with the verify graph vs 26.5 s plain).

Concurrency. Speculating requests are drafted and verified together as one batched round, each with its own persistent draft context, so the gain holds well beyond a single request. Aggregate throughput (tok/s, 512 new tokens, identical prompts, decode CUDA graph on, max-batch-size=48, admit-coalesce-ms=50, BF16 TP=4 on A100 40 GB, default mem-fraction-static=0.85):

Per-request speed at 16 requests is ~26-28 tok/s versus ~21 for plain decode. A batched round of 16 requests takes ~80 ms (draft ~8 ms, verify forward ~52 ms, bookkeeping ~14-19 ms) and commits ~2.5 tokens per request. Outputs match plain decode for the same prompt up to bf16 near-tie flips: plain decode itself is not batch-invariant, and a plain batch of 8 diverges from the same prompts run alone at the same places a speculative batch does.

Memory. Each speculating request needs DeltaNet checkpoint buffers: about 0.04 GB per GPU per draft slot for a 27B hybrid model, i.e. ~0.11 GB at the default depth 3 (~0.075 GB at depth 2), reserved at load for min(max-batch-size, speculative-max-concurrency) rows, before the KV pool is sized, so the KV budget already excludes it (48 rows at depth 3 = ~5.3 GB per GPU on a 27B hybrid model; it shrinks the KV pool, e.g. from ~734k to ~455k token slots on 40 GB GPUs at mem-fraction-static=0.85). Only the slots a partial accept can restore from are stored (positions 1..depth; the pre-window copy and the final position are not needed). A cohort larger than the reservation (only possible after a depth change) is still checked at run time and, if it does not fit, the engine logs one warning and serves it with plain batched decode instead of failing requests. To keep more KV, lower smile.chat.speculative-max-concurrency (smaller reservation) or set smile.chat.speculative=false.

Speculation is on by default for Qwen3.5/3.8 checkpoints with MTP weights; budget the memory above on small GPUs or very large batches, or set smile.chat.speculative=false.

Controls (all default to the values that were benchmarked):

Notes and limits:

• The MTP draft head's KV cache is a second, request-bound pool sized like the main pool (capped at half of the free device memory when it is created, at first use). A request that cannot get one, is multimodal, or takes a plain decode step (for example it holds no speculation slot, or the cohort did not fit in memory) simply continues with plain decode: output is unaffected, it just stops speculating.
• Cohorts of up to 16 speculating requests replay a verify CUDA graph keyed by batch size (rows may sit at different positions; positions, RoPE rows, KV slots and the page table are rewritten in place before each replay, one ~10 ms capture per batch size). Larger cohorts verify eagerly because each captured batch size keeps its own activation pool. Same-session A/B against the eager batched verify (512 tokens): 4 requests 145 → 176 tok/s, 8 requests 255 → 331, 16 requests 461 → 498. The decode CUDA graph is not used while any request is speculating.
• The chat endpoint returns the final answer without the model's hidden reasoning. When comparing a speculative and a plain response at a short max_tokens, differences can appear from the first visible character only because the token limit falls before/after the end of the hidden reasoning; compare at a limit inside the reasoning, or at a long limit.
• Capture happens once per (batch, window length); the 2-token verify window at the very end of a request triggers one extra capture (~0.1 s).
• Troubleshooting: SMILE_DECODE_PROFILE=1 adds a per-round breakdown to the MTP verify: log line; SMILE_VERIFY_CUDA_GRAPH_DEBUG_DIFF=1 (diagnostic, doubles verify cost) logs the numeric gap between graph replay and an eager reference every round; SMILE_VERIFY_GRAPH_TRACE=1 logs each round's replay / warmup / capture decision.

Building and Running

All commands use the Gradle wrapper from the project root.

Dev Mode

Live-reload development mode — changes to Java sources are reflected without restarting. The Quarkus Dev UI is available at http://localhost:8888/q/dev/.

./gradlew :serve:quarkusDev \
  --jvm-args="--add-opens java.base/java.lang=ALL-UNNAMED"

The --add-opens flags are required by ONNX Runtime's Foreign Function Interface. The dev-mode HTTP port defaults to 8888 (configured via %dev.quarkus.http.port).

Packaging as a JAR

./gradlew :serve:build

This produces a Quarkus layered application in build/quarkus-app/. The entry point is build/quarkus-app/quarkus-run.jar; the dependencies live in build/quarkus-app/lib/ and must be distributed together.

Run it with:

java \
  --add-opens java.base/java.lang=ALL-UNNAMED \
  --add-opens java.base/java.nio=ALL-UNNAMED \
  --enable-native-access ALL-UNNAMED \
  -jar build/quarkus-app/quarkus-run.jar

To run on a custom port:

java \
  --add-opens java.base/java.lang=ALL-UNNAMED \
  --add-opens java.base/java.nio=ALL-UNNAMED \
  --enable-native-access ALL-UNNAMED \
  -Dquarkus.http.port=3801 \
  -jar build/quarkus-app/quarkus-run.jar

Uber-JAR

A single self-contained JAR (slower to start, simpler to deploy):

./gradlew :serve:build -Dquarkus.package.jar.type=uber-jar
java \
  --add-opens java.base/java.lang=ALL-UNNAMED \
  --add-opens java.base/java.nio=ALL-UNNAMED \
  --enable-native-access ALL-UNNAMED \
  -jar build/smile-serve-runner.jar

Native Executable

Compile to a native binary with GraalVM (sub-millisecond startup, lower memory):

./gradlew :serve:build -Dquarkus.native.enabled=true
./build/smile-serve-*-runner

Without a local GraalVM installation, use a Docker-based build:

./gradlew :serve:build \
  -Dquarkus.native.enabled=true \
  -Dquarkus.native.container-build=true

See the Quarkus native build guide for details.

Configuration Reference

Configuration is managed in src/main/resources/application.properties. Quarkus profile prefixes (%dev., %test.) override the base values in the corresponding profiles.

Override at runtime with -D system properties, for example:

java ... -Dsmile.serve.model=/data/models/rf_classifier.sml -jar quarkus-run.jar

Classic ML Inference API

Model Format

Classic ML models are serialized Java objects saved in .sml files by the SMILE smile.model.Model framework. They carry:

• The trained algorithm (random forest, SVM, gradient boost, etc.)
• The input feature schema (field names and data types)
• Training / validation metrics
• Optional metadata tags (id, version, user-defined properties)

At startup, InferenceService scans the path specified by the property smile.serve.model. If the path is a regular .sml file only that model is loaded; if it is a directory every .sml file in the directory is loaded.

Get Model Metadata

Returns the algorithm name, input schema, and tags for a model. Use GET /api/v1/models to discover loaded model IDs.

GET /api/v1/smile/{id}

Example:

curl http://localhost:8080/api/v1/smile/iris_random_forest-1

{
  "id": "iris_random_forest-1",
  "algorithm": "random-forest",
  "schema": {
    "petallength": { "type": "float", "nullable": false },
    "petalwidth":  { "type": "float", "nullable": false },
    "sepallength": { "type": "float", "nullable": false },
    "sepalwidth":  { "type": "float", "nullable": false }
  },
  "tags": {
    "smile.random_forest.trees": "200"
  }
}

The schema object lists every input feature in alphabetical order — this is the column order used by the CSV streaming endpoint.

Single Inference (JSON)

POST /api/v1/smile/{id}
Content-Type: application/json

The request body is a flat JSON object whose keys are the feature names defined in the model schema. All non-nullable fields are required.

Classification example (iris):

curl -X POST http://localhost:8080/api/v1/smile/iris_random_forest-1 \
  -H "Content-Type: application/json" \
  -d '{
    "sepallength": 5.1,
    "sepalwidth":  3.5,
    "petallength": 1.4,
    "petalwidth":  0.2
  }'

{
  "prediction": 0,
  "probabilities": [0.960, 0.021, 0.019]
}

• prediction — the predicted class label (integer) or regression value (float).
• probabilities — posterior class probabilities for soft classifiers (e.g. random forest, logistic regression). Absent for hard classifiers and regressors.
• explanations — optional model explanation object (present when requested).

Explanations (TreeSHAP)

Explanations can be requested by passing "enableExplanations": true in the JSON body, or appending ?explain=true to the URL.

curl -X POST "http://localhost:8080/api/v1/smile/iris_random_forest-1?explain=true" \
  -H "Content-Type: application/json" \
  -d '{
    "sepallength": 5.1,
    "sepalwidth":  3.5,
    "petallength": 1.4,
    "petalwidth":  0.2
  }'

{
  "prediction": 0,
  "probabilities": [0.960, 0.021, 0.019],
  "explanations": {
    "shap": [
      [0.012, -0.045, 0.123, -0.002],
      [-0.034, 0.056, -0.210, 0.015],
      [0.022, -0.011, 0.087, -0.013]
    ]
  }
}

• For tree classification models (Random Forest, AdaBoost, Gradient Tree Boost, Decision Tree), shap is a 2D array of floats with dimension [classes][features], formatted to 3 decimal places.
• For tree regression models, shap is a 1D array of floats with dimension [features].
• For models that do not support SHAP, shap returns "Not supported".

Error responses:

Streaming Inference (CSV / JSON-lines)

Process many samples in a single request. The server returns results as a Server-Sent Events stream emitting structured JSON objects — one data: line per input sample. Append ?explain=true to include explanations in each streaming event.

POST /api/v1/smile/{id}/stream
Content-Type: text/plain          ← CSV mode
Content-Type: application/json   ← JSON-lines mode

CSV mode (text/plain)

Each non-blank line is a comma-separated row of feature values in the same column order as the model schema (alphabetical by field name, as shown by GET /api/v1/smile/{id}).

cat iris.csv | curl -X POST \
  -H "Content-Type: text/plain" \
  --data-binary @- \
  http://localhost:8080/api/v1/smile/iris_random_forest-1/stream

Where iris.csv might contain:

5.1,3.5,1.4,0.2
6.7,3.0,5.2,2.3
5.8,2.7,4.1,1.0

The response stream (SSE format emitting JSON payloads):

data: {"prediction":0,"probabilities":[0.960,0.021,0.019]}

data: {"prediction":2,"probabilities":[0.012,0.051,0.937]}

data: {"prediction":1,"probabilities":[0.031,0.752,0.217]}

JSON-lines mode (application/json)

Each non-blank line must be a complete JSON object (one per line). This is more verbose but supports named fields in any order. Each sample can also independently include "enableExplanations": true.

cat iris.jsonl | curl -X POST \
  -H "Content-Type: application/json" \
  --data-binary @- \
  http://localhost:8080/api/v1/smile/iris_random_forest-1/stream

Where iris.jsonl contains:

{"sepallength":5.1,"sepalwidth":3.5,"petallength":1.4,"petalwidth":0.2}
{"sepallength":6.7,"sepalwidth":3.0,"petallength":5.2,"petalwidth":2.3,"enableExplanations":true}

Model IDs

A model's ID is constructed as <name>-<version> from the model's embedded metadata tags (smile.model.Model.ID and smile.model.Model.VERSION). If those tags are absent, the file name stem is used as the name and "1" as the version. For example, a file named iris_random_forest.sml with no ID tag gets the ID iris_random_forest-1.

ONNX Inference API

The ONNX endpoint exposes any model in the ONNX open format through SMILE's native ONNX Runtime binding (smile.onnx). This covers models exported from PyTorch, TensorFlow, scikit-learn (via sklearn-onnx), and many other frameworks. Classical ONNX inference is documented on ONNX.

Model Format

At startup, OnnxService scans the folder specified by the property smile.onnx.model. Every .onnx file found is loaded into an InferenceSession. The model ID is the file name without the .onnx extension (e.g., resnet50.onnx → ID resnet50).

Native library: the ONNX Runtime shared library (libonnxruntime.so / libonnxruntime.dylib / onnxruntime.dll) must be on the OS library search path, and the JVM must include --enable-native-access=ALL-UNNAMED. If the library is missing, startup continues without ONNX models (endpoints return 404) rather than aborting the whole serve process. Download pre-built binaries from the ORT releases page.

Get ONNX Model Info

Returns graph metadata and the typed, shaped input/output node descriptors. Use GET /api/v1/models to discover loaded model IDs.

GET /api/v1/onnx/{id}

curl http://localhost:8080/api/v1/onnx/resnet50

{
  "id": "resnet50",
  "graphName": "ResNet50",
  "description": "Image classification model",
  "version": 1,
  "inputs": [
    {
      "name": "input",
      "onnxType": "TENSOR",
      "elementType": "FLOAT",
      "shape": [1, 3, 224, 224]
    }
  ],
  "outputs": [
    {
      "name": "output",
      "onnxType": "TENSOR",
      "elementType": "FLOAT",
      "shape": [1, 1000]
    }
  ],
  "customMeta": {}
}

A shape value of -1 means that dimension is dynamic (determined at inference time from the input data).

Single Inference (JSON)

POST /api/v1/onnx/{id}
Content-Type: application/json

The request body is a JSON object mapping each input name to a flat JSON array of numbers or strings. The server constructs the required ORT tensor from the declared element type and shape.

Example — image classification (resnet50, 1×3×224×224 = 150528 floats):

curl -X POST http://localhost:8080/api/v1/onnx/resnet50 \
  -H "Content-Type: application/json" \
  -d '{"input": [0.485, 0.456, 0.406, ...]}'

The Infer UI image path resizes to the model's H×W, scales to [0, 1], then applies ImageNet channel normalization (x - mean) / std with mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] (NCHW or NHWC as declared). When posting tensors yourself, use the same layout and normalization the model was trained with.

Response — a JSON object mapping each output name to a flat array of raw logits (not probabilities) or strings. ImageNet classifiers from the ONNX Model Zoo typically omit a Softmax node; apply MathEx.softmax before interpreting scores as class probabilities (same as InferenceSessionTest in core).

{
  "output": [0.001, 0.002, 0.872, 0.003, ...]
}

Multi-input model example:

curl -X POST http://localhost:8080/api/v1/onnx/bert_classifier \
  -H "Content-Type: application/json" \
  -d '{
    "input_ids":      [101, 2054, 2003, 1996, 3007, 1997, 2605, 1029, 102],
    "attention_mask": [1,   1,    1,    1,    1,    1,    1,    1,    1  ],
    "token_type_ids": [0,   0,    0,    0,    0,    0,    0,    0,    0  ]
  }'

String input model example (e.g. scikit-learn pipeline with categorical string features or text inputs):

curl -X POST http://localhost:8080/api/v1/onnx/pipeline_classifier \
  -H "Content-Type: application/json" \
  -d '{
    "category": ["electronics", "apparel"],
    "rating":   [4.5, 3.0]
  }'

Supported input element types:

Error responses:

Streaming Inference

Identical in structure to the classic ML streaming endpoint but returns JSON objects:

POST /api/v1/onnx/{id}/stream
Content-Type: text/plain          ← CSV floats for single-input models
Content-Type: application/json   ← JSON-lines for multi-input models

CSV (single-input models only):

cat features.csv | curl -X POST \
  -H "Content-Type: text/plain" \
  --data-binary @- \
  http://localhost:8080/api/v1/onnx/my_classifier/stream

Each response line is a compact JSON object:

data: {"output":[0.02,0.95,0.03]}

data: {"output":[0.88,0.07,0.05]}

JSON-lines (any number of inputs):

cat samples.jsonl | curl -X POST \
  -H "Content-Type: application/json" \
  --data-binary @- \
  http://localhost:8080/api/v1/onnx/bert_classifier/stream

Tensor Types and Shape Resolution

The server automatically resolves the ORT tensor shape from the model's declared input shape and the actual array length:

• Fully static shape (no -1 dimensions) — the array length must exactly match the product of all dimensions. A mismatch returns HTTP 400.
• Single dynamic dimension — the unknown dimension is inferred as arrayLength / product(staticDimensions). For example, a declared shape [-1, 3, 224, 224] with 150528 elements resolves to [1, 3, 224, 224].
• Multiple dynamic dimensions — the shape is set to [1, arrayLength].
• No shape info — the shape is set to [1, arrayLength].

LLM Chat API

SMILE Serve includes a Java implementation of Large Language Models for on-premise LLM inference. The chat API is designed to be compatible with the OpenAI Chat Completions interface.

The LLM is optional: if the path specified by the property smile.chat.model does not exist on the file system (and cannot be resolved via Hugging Face / OGA), ChatService starts in an unavailable state and every request to the chat endpoints returns HTTP 503 Service Unavailable.

Backend selection: with CUDA and a builtin Llama/Qwen checkpoint, serve uses the Torch path + continuous batching. Otherwise it falls back to ONNX Runtime GenAI (GenAiChatModel) when the model is already GenAI-ready (genai_config.json at the repo root or under provider folders such as cuda/cuda-int4-…/, or a prior Olive cache under {SMILE_CACHE}/olive). Serve does not run Olive at startup — convert offline with smile.chat.Olive (see smile.chat.oga.*). Tools input / structured tool_calls output work on the serial GenAI path; multimodal GenAI materializes media to temp files. See ONNX GenAI.

There are therefore two ways to run a chat model — the native (LibTorch) path for the built-in Llama-3.1 and Qwen3.5/3.8 implementations, and the ONNX Runtime GenAI path for every other family packaged as a GenAI-ready directory.

Supported Models

The native path covers Llama-3.1 and Qwen3.5/3.8 checkpoints. Every other family is served through ONNX Runtime GenAI and needs only a GenAI-ready package — published on the Hugging Face Hub, converted offline with Olive (smile.chat.Olive, see ONNX GenAI), or built with the ONNX Runtime GenAI model builder. Pass any repo id below (or a local directory) to smile.chat.model:

./gradlew :serve:quarkusDev "-Dsmile.chat.model=microsoft/Phi-4-mini-instruct-onnx"

Language models (decoder-only). The runtime dispatches on the architecture recorded in genai_config.json; the checkpoint listed for each family is a known-good, GenAI-ready Hub repository.

Vision-language models. GenAiChatModel accepts local image/audio paths through MultiModalProcessor; serve materializes data-URL / HTTP media to temp files first.

Audio models. Whisper speech recognition is supported by the runtime; select it outside the chat API (the chat endpoint is text-only).

The architecture list mirrors the upstream ONNX Runtime GenAI supported models and model-builder support matrix; the checkpoint ids are Hub repositories that ship genai_config.json. File layouts and provider folders vary per repo — GenAiModelPaths auto-detects the nested package (CUDA / DirectML / CPU) that matches your execution provider.

List models

GET /api/v1/models

OpenAI-compatible catalog of all loaded models — chat LLMs, ONNX graphs, and SMILE .sml models. Inference still uses type-specific paths: /api/v1/chat/completions, /api/v1/onnx/{id}, /api/v1/smile/{id}.

curl http://localhost:8080/api/v1/models

{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "object": "model",
      "created": 1741900000,
      "owned_by": "meta-llama",
      "shutdown_date": null,
      "kind": "LLM"
    },
    {
      "id": "iris_random_forest-1",
      "object": "model",
      "created": 1710000000,
      "owned_by": "Unknown",
      "shutdown_date": null,
      "kind": "random-forest"
    },
    {
      "id": "resnet50",
      "object": "model",
      "created": 1710000000,
      "owned_by": "Unknown",
      "shutdown_date": null,
      "kind": "ONNX"
    }
  ]
}

kind values:

• LLM — chat / completion models
• ONNX — ONNX Runtime graphs
• SMILE algorithm name — e.g. random-forest, cart, logistic (from the .sml model)

owned_by rules:

• Chat (HF): first segment of the repo id (meta-llama/... → meta-llama)
• Chat (local): first segment of Llama.family() (meta/llama3 → meta)
• SMILE .sml: tag author, else owner; otherwise Unknown
• ONNX: custom metadata author/owner when present; otherwise Unknown

Retrieve model

GET /api/v1/models/{id}

OpenAI-compatible retrieve model. Returns the same base ModelObject fields as list entries, plus an optional type-specific detail block. Does not run inference — use /chat/completions, /onnx/{id}, or /smile/{id} for that. Ids may contain slashes (e.g. Hugging Face repo ids).

List responses omit smile / onnx / llm so the catalog stays lean.

curl http://localhost:8080/api/v1/models/iris_random_forest-1

{
  "id": "iris_random_forest-1",
  "object": "model",
  "created": 1710000000,
  "owned_by": "Unknown",
  "shutdown_date": null,
  "kind": "random-forest",
  "smile": {
    "formula": "class ~ .",
    "schema": {
      "petallength": { "type": "float", "nullable": false }
    },
    "tags": {},
    "train": {
      "accuracy": 0.97,
      "size": 150
    },
    "validation": null,
    "test": null
  }
}

Chat Completions

POST /api/v1/chat/completions
Content-Type: application/json

Tokens are streamed back as Server-Sent Events when stream is true, or returned as a single OpenAI chat.completion JSON object when stream is false or omitted (OpenAI default). The conversation (user message + assistant reply) is automatically persisted to the configured database after generation finishes.

Serve runs an InferenceEngine continuous-batching loop (Fluid Injection admission, chunked prefill, batched decode, Instant Eviction on abort). Offline library callers may still use LanguageModel.generate; HTTP chat always goes through the engine. Client disconnect on an SSE stream aborts the GenerationHandle, which Instant-Evicts that request's KV pages without stopping peer in-flight generations.

Request body fields (snake_case):

Each Message has a role (system, user, assistant, or tool) and content. Assistant turns may include tool_calls; tool-result turns use role: "tool" and tool_call_id. Legacy "ipython" is accepted as an alias for "tool".

Tool calling (Qwen3 / Qwen3.5 / Qwen3.8)

OpenAI-compatible tool calling is supported for latest Qwen checkpoints (XML <tool_call> format). Clients run tools locally and continue the dialog:

curl -X POST http://localhost:8080/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is the weather in SF?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_current_temperature",
        "description": "Get temperature for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

A successful tool-call response uses finish_reason: "tool_calls" and message.tool_calls[]. Append tool results as {"role":"tool","tool_call_id":"…","content":"…"} and call again for the final answer. Streaming (v1) buffers the completion and replays delta.tool_calls at the end (not token-true XML streaming).

Smoke test with the OpenAI Python SDK:

python serve/scripts/test_tool_calling.py --base-url http://localhost:8888/api/v1

Streaming example (stream: true):

curl -X POST http://localhost:8080/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "stream": true,
    "messages": [
      {"role": "system",  "content": "You are a helpful assistant."},
      {"role": "user",    "content": "What is the capital of France?"}
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

The response is an SSE stream of OpenAI-shaped chat.completion.chunk JSON events, terminated by data: [DONE].

Non-streaming example (stream omitted or false):

curl -X POST http://localhost:8080/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_completion_tokens": 256
  }'

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1741900000,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Paris." },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 3,
    "total_tokens": 23
  }
}

Example — continue a previous conversation:

curl -X POST http://localhost:8080/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -N \
  -d '{
    "conversation": 42,
    "messages": [
      {"role": "user", "content": "What about Germany?"}
    ]
  }'

Conversation History API

Chat history is stored in a relational database (PostgreSQL in production, H2 in dev mode). The API base path is /api/v1/conversations.

Conversations are scoped to authenticated users. Guests (no session) may still create conversations and chat, but those rows are tied to client IP and do not appear in the sidebar list until the user signs in (recent guest threads from the same IP within 24 hours are merged on login).

On localhost or 127.0.0.1 (any port), the server auto-logs you in as the local me account so dev workflows work without OAuth.

Authentication

The authentication endpoints manage user identity specifically for conversation history and personal settings. Note that this is a session-cookie identity mechanism intended for personal use and UI convenience, not an enterprise authentication or authorization system: enterprise access control belongs to the API gateway layer, and ML/ONNX inference endpoints are intentionally unauthenticated at the engine level.

Configure Google login in application.properties (or env):

smile.auth.google.client-id=${GOOGLE_CLIENT_ID}
smile.auth.google.client-secret=${GOOGLE_CLIENT_SECRET}
# Optional; default is {origin}/api/v1/auth/callback/google
# smile.auth.google.redirect-uri=https://your-host/api/v1/auth/callback/google
smile.auth.session-secret=${SMILE_SESSION_SECRET}

Register the redirect URI in Google Cloud Console. Personal instructions from the user profile are injected as the chat system prompt when logged in.

Create, retrieve, update, and delete follow the OpenAI Conversations shapes. Conversation ids are strings of the form conv_<n>. List, PATCH (title/pin), and GET .../items are SMILE extensions.

List conversations (SMILE extension, requires login)

GET /api/v1/conversations?pageIndex=0&pageSize=25&q=search&pinned=true

Returns the signed-in user's conversations (title, pinned, updated_at). Search matches title and message content.

curl "http://localhost:8080/api/v1/conversations?pageSize=10"

Create a conversation

POST /api/v1/conversations
Content-Type: application/json

Optional body fields: metadata (≤16 string pairs) and items (≤20 message items with role + text content).

curl http://localhost:8080/api/v1/conversations \
  -H "Content-Type: application/json" \
  -d '{"metadata":{"topic":"demo"},"items":[{"type":"message","role":"user","content":"Hello!"}]}'

{
  "id": "conv_1",
  "object": "conversation",
  "created_at": 1741900000,
  "metadata": {"topic": "demo"}
}

Retrieve a conversation

GET /api/v1/conversations/{conversation_id}

Update a conversation

POST /api/v1/conversations/{conversation_id}
Content-Type: application/json

curl http://localhost:8080/api/v1/conversations/conv_1 \
  -H "Content-Type: application/json" \
  -d '{"metadata":{"topic":"project-x"}}'

Patch sidebar fields (SMILE extension, requires ownership)

PATCH /api/v1/conversations/{conversation_id}
Content-Type: application/json

{"title": "My chat", "pinned": true}

Delete a conversation

DELETE /api/v1/conversations/{conversation_id}

{
  "id": "conv_1",
  "object": "conversation.deleted",
  "deleted": true
}

Get conversation messages (SMILE extension)

GET /api/v1/conversations/{conversation_id}/items?pageIndex=0&pageSize=25

Returns the individual message turns (role + content + createdAt) in chronological order.

curl http://localhost:8080/api/v1/conversations/conv_42/items

[
  { "id": 1, "conversationId": 42, "role": "user",      "content": "What is the capital of France?", "createdAt": "2026-04-15T10:00:00Z" },
  { "id": 2, "conversationId": 42, "role": "assistant", "content": "The capital of France is Paris.", "createdAt": "2026-04-15T10:00:02Z" }
]

Web UI

A React-based web interface is bundled via Quarkus Quinoa. It is served from the root URL and provides:

• Inference UI (/infer) — unified model shell: collapsible sidebar lists chat, SMILE (.sml), and ONNX (.onnx) models. Selecting a chat model embeds the chat module with a collapsible right nav (history, login, settings). SMILE models get a schema-driven form; ONNX models get a numeric form from tensor shapes, or an image upload when a 4-D vision-like input is detected (overrideable).
• Chat UI (/chat) — standalone chat with the same right nav panel (New Chat always available; history/login/settings when signed in). Streaming tokens, Markdown/math. On localhost, you are auto-signed-in as me.

In dev mode the React development server runs on port 5173 and requests are proxied to the Quarkus backend. The production build (dist/) is served statically by the Quarkus process.

Database

Chat conversation history requires a relational database.

To enable the database in production set:

quarkus.hibernate-orm.active=true
quarkus.datasource.username=<user>
quarkus.datasource.password=<password>

Hibernate ORM uses drop-and-create by default. Change the strategy in production to update or validate:

quarkus.hibernate-orm.schema-management.strategy=update

The database is not required for the ML or ONNX inference endpoints — only for chat conversation persistence.

Testing

./gradlew :serve:test

The test profile (%test.*) configures the service with:

• An in-memory H2 database (no external database required).
• A pre-trained iris random forest model from serve/src/test/resources/model/iris_random_forest.sml.
• The ONNX model path also pointed at the test resources directory (no .onnx files present by default, so OnnxService starts empty).
• The chat model path set to a non-existent path so ChatService starts gracefully unavailable without attempting to load a GPU model.

The test class InferenceResourceTest covers:

API Quick Reference

Classic ML — /api/v1/smile

ONNX — /api/v1/onnx

Chat — /api/v1/models, /api/v1/chat, /api/v1/conversations

See Also

• Large Language Models — LanguageModel API and LibTorch path
• ONNX — classical InferenceSession / execution providers
• ONNX GenAI — ORT GenAI bindings and OGA fallback
• Deep Learning — neural network training and inference

For the absolute latest details, see the SMILE Serve user guide in the repository.