Class GenerationHandle

java.lang.Object

smile.llm.engine.GenerationHandle

public final class GenerationHandle
extends Object

Handle for an in-flight or queued GenerationRequest.

Method Summary

Modifier and Type	Method	Description
void	abort()	Requests cancellation.
CompletableFuture<ChatCompletion>	future()	Returns the completion future for this generation.
boolean	isAborted()	Returns whether this handle was aborted.
static GenerationHandle	of(long requestId, CompletableFuture<ChatCompletion> future)	Creates a handle for a future owned outside InferenceEngine (e.g.
long	requestId()	Returns the engine-assigned request id.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

of

public static GenerationHandle of(long requestId,
 CompletableFuture<ChatCompletion> future)

Creates a handle for a future owned outside InferenceEngine (e.g. serve fallback when no engine is present).

Parameters:

requestId - engine-assigned request id.

future - completion future for the generation.

Returns:

new handle wrapping future.

requestId

public long requestId()

Returns the engine-assigned request id.

Returns:

request id.

future

public CompletableFuture<ChatCompletion> future()

Returns the completion future for this generation.

Returns:

completion of the generation (exceptionally if aborted or failed).

abort

public void abort()

Requests cancellation. Queued jobs are dropped (Instant Eviction from the wait queue). Running jobs stop cooperatively between decode steps when the model sees isAborted() (at most one forward after abort).

isAborted

public boolean isAborted()

Returns whether this handle was aborted.

Returns:

true after abort().