Class InferenceEngine

java.lang.Object
smile.llm.engine.InferenceEngine
All Implemented Interfaces:
AutoCloseable
public final class InferenceEngine extends Object implements AutoCloseable
Continuous-batching inference runtime.

Fluid Injection admits work up to maxInFlight while KV pages are free. Each tick prefills (optionally chunked under a token budget), then runs a batched ModelExecutor.decodeStep(int[], int[], int[]) over all decoding requests. GenerationHandle.abort() Instant-Evicts queued and in-flight KV.

Optional idle admit coalesce (admitCoalesceMs) delays the first admission after idle so a burst can form a uniform decode cohort; admission proceeds early once queued >= maxInFlight.

When ModelExecutor.supportsStepApi() is false (test stubs), the engine falls back to serial LanguageModel.generate(int[], int, double, double, boolean, long, GenerationListener).

  • Field Details

    • DEFAULT_PREFILL_TOKEN_BUDGET

      public static final int DEFAULT_PREFILL_TOKEN_BUDGET
      Default max new-prefill tokens per scheduler tick.
      See Also:

    • DEFAULT_ADMISSION_TIMEOUT_MS

      public static final long DEFAULT_ADMISSION_TIMEOUT_MS
      Default max time a job may wait for admission before failing.
      See Also:

  • Constructor Details

    • InferenceEngine

      public InferenceEngine(ModelExecutor executor, int maxInFlight)
      Creates an inference engine with default prefill budget and admission timeout.
      Parameters:
      executor - model execution surface.
      maxInFlight - max concurrent admitted jobs (smile.chat.max-batch-size).

    • InferenceEngine

      public InferenceEngine(ModelExecutor executor, int maxInFlight, int maxDecodeBatch, int prefillTokenBudget, long admissionTimeoutMs)
      Creates an inference engine with explicit batching budgets.
      Parameters:
      executor - model execution surface.
      maxInFlight - Fluid Injection cap.
      maxDecodeBatch - max requests in one decodeStep (<= maxInFlight).
      prefillTokenBudget - max prompt tokens prefilled per tick.
      admissionTimeoutMs - fail waiting jobs after this many ms (<= 0 = never).

    • InferenceEngine

      public InferenceEngine(ModelExecutor executor, int maxInFlight, int maxDecodeBatch, int prefillTokenBudget, long admissionTimeoutMs, long admitCoalesceMs)
      Creates an inference engine with explicit batching and admit-coalesce settings.
      Parameters:
      executor - model execution surface.
      maxInFlight - Fluid Injection cap.
      maxDecodeBatch - max requests in one decodeStep (<= maxInFlight).
      prefillTokenBudget - max prompt tokens prefilled per tick.
      admissionTimeoutMs - fail waiting jobs after this many ms (<= 0 = never).
      admitCoalesceMs - when idle, wait up to this many ms for a fuller queue before the first admit (0 = off). Stops early if queued >= maxInFlight.

  • Method Details

    • model

      public LanguageModel model()
      Returns the underlying language model.
      Returns:
      underlying language model.

    • maxInFlight

      public int maxInFlight()
      Returns the max in-flight generations.
      Returns:
      Fluid Injection cap.

    • maxDecodeBatch

      public int maxDecodeBatch()
      Returns the max decode batch size per step.
      Returns:
      max decode batch size.

    • admitCoalesceMs

      public long admitCoalesceMs()
      Idle admit-coalesce window in milliseconds (0 = disabled). Property: smile.chat.admit-coalesce-ms.
      Returns:
      admit coalesce window in milliseconds.

    • maxSpeculativeConcurrency

      public int maxSpeculativeConcurrency()
      Returns the max concurrently-decoding speculative requests before a tick falls back to plain batched decode. Property: smile.chat.speculative-max-concurrency.
      Returns:
      max speculative concurrency.

    • setMaxSpeculativeConcurrency

      public void setMaxSpeculativeConcurrency(int maxSpeculativeConcurrency)
      Sets the max concurrently-decoding speculative requests before a tick falls back to plain batched decode for all speculative-flagged requests. Lower this if per-request eager draft/verify rounds cost more than the batching they give up at your measured concurrency; raise it if your workload shows speculative decoding still wins at higher concurrency.
      Parameters:
      maxSpeculativeConcurrency - max speculative concurrency (>= 1).

    • queueSize

      public int queueSize()
      Returns the number of jobs waiting for admission.
      Returns:
      queued job count.

    • inFlight

      public int inFlight()
      Returns the number of jobs admitted / running on the worker.
      Returns:
      in-flight job count.

    • kvFreeSlots

      public int kvFreeSlots()
      Returns free KV slots when a pool is present.
      Returns:
      free KV slots, or -1 if none.

    • kvFreePages

      public int kvFreePages()
      Returns free KV pages when a pool is present.
      Returns:
      free KV pages, or -1 if none.

    • kvCachePool

      public KvCachePool kvCachePool()
      Returns the primary KV cache pool, or null when the model has no full-attention layers. Exposed for observability (KV usage / prefix-cache counters).
      Returns:
      KV cache pool, or null.

    • activeDecodeCount

      public int activeDecodeCount()
      Returns active decode requests on the last tick (approx).
      Returns:
      approximate active decode count.

    • queueWaitMsTotal

      public long queueWaitMsTotal()
      Returns cumulative queue wait milliseconds.
      Returns:
      total queue wait milliseconds.

    • prefillMsTotal

      public long prefillMsTotal()
      Returns cumulative prefill milliseconds.
      Returns:
      total prefill milliseconds.

    • decodeMsTotal

      public long decodeMsTotal()
      Returns cumulative decode-step milliseconds.
      Returns:
      total decode-step milliseconds.

    • submit

      public GenerationHandle submit(GenerationRequest request)
      Enqueues a generation request.
      Parameters:
      request - generation parameters.
      Returns:
      handle with future + abort.

    • submit

      public GenerationHandle submit(GenerationRequest request, Consumer<GenerationHandle> onCreated)
      Enqueues a generation request.

      onCreated runs after the handle exists and before the job is visible to the worker (so listeners can bind requestId without racing the first decode tokens).

      Parameters:
      request - generation parameters.
      onCreated - callback with the new handle; must not block.
      Returns:
      handle with future + abort.

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable