Interface GenerationListener

public interface GenerationListener
Callback for autoregressive generation progress on a single request.

Serve and other clients implement this (or compose helpers) instead of passing a SubmissionPublisher into the model. A streaming adapter can forward onText(String) to a publisher; a metrics sink can override only the counters it cares about. When many requests run concurrently, the InferenceEngine binds one listener per request.

Typical order for one request:

  1. onInputTokens(int) / onCachedInputTokens(int) once after prompt bind
  2. repeated onGeneratedTokens(int) (and optional onText(String))
  3. onThinkingTokens(int) when the engine can attribute “thinking” tokens (not emitted yet)

  • Method Summary

    Modifier and Type
    Method
    Description
    default void
    onCachedInputTokens(int count)
    Reports how many input tokens were served from the KV prefix cache (radix hit), so they need not be recomputed in prefill.
    default void
    onGeneratedTokens(int count)
    Reports newly generated completion tokens (not prompt tokens).
    default void
    onInputTokens(int count)
    Reports the number of prompt / input tokens.
    default void
    onText(String chunk)
    Reports a decoded UTF-8 text chunk suitable for streaming to a client.
    default void
    onThinkingTokens(int count)
    Reports tokens attributed to a model “thinking” / reasoning span (e.g.

  • Method Details

    • onInputTokens

      default void onInputTokens(int count)
      Reports the number of prompt / input tokens.
      Parameters:
      count - prompt token count; >= 0.

    • onCachedInputTokens

      default void onCachedInputTokens(int count)
      Reports how many input tokens were served from the KV prefix cache (radix hit), so they need not be recomputed in prefill.

      Always <= the value passed to onInputTokens(int). When prefix reuse is disabled or there is no match, count is 0.

      Parameters:
      count - cached input token count; >= 0.

    • onGeneratedTokens

      default void onGeneratedTokens(int count)
      Reports newly generated completion tokens (not prompt tokens).
      Parameters:
      count - number of tokens produced since the previous call; typically 1.

    • onThinkingTokens

      default void onThinkingTokens(int count)
      Reports tokens attributed to a model “thinking” / reasoning span (e.g. Qwen thinking mode), distinct from final answer tokens.

      Reserved: the engine does not emit this yet. Implementations may override it for forward compatibility; the default is a no-op.

      Parameters:
      count - thinking tokens since the previous call; >= 0.

    • onText

      default void onText(String chunk)
      Reports a decoded UTF-8 text chunk suitable for streaming to a client.

      May be called less often than onGeneratedTokens(int) (implementations commonly coalesce ~20 tokens, or flush on EOS). Empty strings are not delivered.

      Parameters:
      chunk - decoded text (special tokens already skipped when applicable).