Class Fp8KvCodec

java.lang.Object

smile.llm.quant.Fp8KvCodec

public final class Fp8KvCodec
extends Object

Helpers for FP8 KV store/load with per-tensor scales.

Used by KvCachePool when smile.chat.kv-cache.dtype is FP8. Ampere stores FP8 and dequants to BF16/FP16 for attention compute; Hopper may later use native FlashInfer FP8.

Field Summary

Fields

Modifier and Type	Field	Description
static final float	E4M3_MAX	Max finite magnitude for Float8 e4m3fn.

Method Summary

Modifier and Type	Method	Description
static float	computeScale(Tensor x, float fp8Max)	Computes a positive float scale so abs(x)/scale <= fp8Max.
static Tensor	dequantize(Tensor q, float scale, ScalarType outDtype)	Dequantizes FP8 q with scale to outDtype.
static boolean	isFp8(ScalarType dtype)	Returns true when dtype is an FP8 scalar type.
static Tensor	quantize(Tensor x, float scale, ScalarType fp8Dtype)	Quantizes x to FP8 using scale (x_fp8 ≈ x / scale).

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

E4M3_MAX

public static final float E4M3_MAX

Max finite magnitude for Float8 e4m3fn.

See Also:

• Constant Field Values

Method Details

isFp8

public static boolean isFp8(ScalarType dtype)

Returns true when dtype is an FP8 scalar type.

Parameters:

dtype - scalar type to test.

Returns:

true when dtype is FP8.

computeScale

public static float computeScale(Tensor x,
 float fp8Max)

Computes a positive float scale so abs(x)/scale <= fp8Max.

Parameters:

x - tensor to measure.

fp8Max - max finite FP8 magnitude.

Returns:

positive scale for quantization.

quantize

public static Tensor quantize(Tensor x,
 float scale,
 ScalarType fp8Dtype)

Quantizes x to FP8 using scale (x_fp8 ≈ x / scale).

Parameters:

x - source tensor.

scale - positive float scale.

fp8Dtype - target FP8 dtype.

Returns:

quantized FP8 tensor (caller owns).

dequantize

public static Tensor dequantize(Tensor q,
 float scale,
 ScalarType outDtype)

Dequantizes FP8 q with scale to outDtype.

Parameters:

q - FP8 tensor.

scale - positive float scale.

outDtype - output compute dtype.

Returns:

dequantized tensor (caller owns).