Class VerifyCudaGraph

java.lang.Object
smile.llm.engine.VerifyCudaGraph
public final class VerifyCudaGraph extends Object
Optional CUDA graph capture for the multi-token MTP window-verify forward (S = numDrafts + 1), independent of DecodeCudaGraph.

Enable with environment variable SMILE_VERIFY_CUDA_GRAPH=1 — this both wires the new stable-buffer / graph-capturable-kernel path and enables real capture (captureEnabled()), mirroring SMILE_DECODE_CUDA_GRAPH's single-flag design. Stage 4 of the verify-CUDA-graph plan validated the buffer/kernel path in a shadow-run mode with capture hard-disabled (a now-removed compile-time constant, not a runtime flag); Stage 5 enables real capture once Stage 4's numerics were validated against the existing eager path on real hardware (byte-identical output, no perf regression).

Bucketed by (batch, windowLen, numPages), mirroring DecodeCudaGraph's (batch, numPages) bucketing plus the extra windowLen dimension (query length is not implicitly 1 here).

  • Method Summary

    Modifier and Type
    Method
    Description
    static boolean
    canGraphVerify(int[] startPositions)
    Returns whether this verify step is eligible for CUDA graph capture/replay.
    static boolean
    captureEnabled()
    Returns whether real graph capture may be attempted (Stage 5: same gate as enabled() — a single flag, mirroring SMILE_DECODE_CUDA_GRAPH).
    static boolean
    debugDiff()
    Returns whether the diagnostic eager-vs-graph verify diff is enabled (see DEBUG_DIFF).
    static void
    disableCapture(String reason)
    Permanently disable verify CUDA graphs for this process (after a capture failure).
    static boolean
    enabled()
    Returns whether verify CUDA-graph buffer/kernel wiring is enabled for this process (independent of whether capture itself is enabled).
    static void
    markPersistentLogits(boolean persistent)
    Marks that the current verify step returned logits backed by a captured CUDA graph output buffer (must not be closed by the caller) — mirrors DecodeCudaGraph.markPersistentLogits(boolean).
    static int
    maxBatch()
    Largest batch that uses a verify graph (each captured batch size keeps its own activation pool, which at large batches competes with the DeltaNet checkpoint buffers for GPU memory).
    static boolean
    persistentLogits()
    Returns whether verify logits outlive the caller's own scope (i.e.
    static boolean
    raggedEnabled()
    Whether ragged (per-row positions, batch > 1) windows may use the verify graph.
    static int
    warmupSteps()
    Returns the number of eager warmup verify steps before graph capture.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • enabled

      public static boolean enabled()
      Returns whether verify CUDA-graph buffer/kernel wiring is enabled for this process (independent of whether capture itself is enabled).
      Returns:
      true when env is set, native API is linked, and capture is not disabled.

    • captureEnabled

      public static boolean captureEnabled()
      Returns whether real graph capture may be attempted (Stage 5: same gate as enabled() — a single flag, mirroring SMILE_DECODE_CUDA_GRAPH).
      Returns:
      true when verify CUDA graphs are enabled and capture has not been disabled after a prior failure.

    • raggedEnabled

      public static boolean raggedEnabled()
      Whether ragged (per-row positions, batch > 1) windows may use the verify graph. On by default with the verify graph; SMILE_VERIFY_CUDA_GRAPH_RAGGED=0 disables.
      Returns:
      true when ragged batched graphs are enabled.

    • maxBatch

      public static int maxBatch()
      Largest batch that uses a verify graph (each captured batch size keeps its own activation pool, which at large batches competes with the DeltaNet checkpoint buffers for GPU memory). Override with SMILE_VERIFY_CUDA_GRAPH_MAX_BATCH.
      Returns:
      maximum batch size eligible for graph capture.

    • canGraphVerify

      public static boolean canGraphVerify(int[] startPositions)
      Returns whether this verify step is eligible for CUDA graph capture/replay.
      Parameters:
      startPositions - per-row KV window start positions.
      Returns:
      true when every row shares the same start position (a homogeneous cohort; batch > 1 verify-graph is otherwise out of scope today, but the check does not hardcode batch == 1).

    • disableCapture

      public static void disableCapture(String reason)
      Permanently disable verify CUDA graphs for this process (after a capture failure).
      Parameters:
      reason - human-readable failure reason (for callers; not logged here).

    • warmupSteps

      public static int warmupSteps()
      Returns the number of eager warmup verify steps before graph capture.
      Returns:
      warmup steps per (batch, windowLen, numPages) bucket.

    • debugDiff

      public static boolean debugDiff()
      Returns whether the diagnostic eager-vs-graph verify diff is enabled (see DEBUG_DIFF).
      Returns:
      true when SMILE_VERIFY_CUDA_GRAPH_DEBUG_DIFF=1.

    • markPersistentLogits

      public static void markPersistentLogits(boolean persistent)
      Marks that the current verify step returned logits backed by a captured CUDA graph output buffer (must not be closed by the caller) — mirrors DecodeCudaGraph.markPersistentLogits(boolean). A shared AtomicBoolean, not thread-local: TP fan-out sets this from each rank's worker thread, and the join point (after every rank's future completes) reads it back on the calling thread.
      Parameters:
      persistent - true when logits outlive the caller's own scope.

    • persistentLogits

      public static boolean persistentLogits()
      Returns whether verify logits outlive the caller's own scope (i.e. are QwenModel's own reused capture buffer, not a throwaway eager tensor).
      Returns:
      true when the caller must not close the returned tensor.