Class QuantizedHfLoader

java.lang.Object

smile.llm.quant.QuantizedHfLoader

public final class QuantizedHfLoader
extends Object

Loads HuggingFace GPTQ / AWQ / FP8 / NVFP4 SafeTensors into Llama linears (shard-then-pack for TP when tpSize > 1).

Marlin path: batch-reads each safetensors shard once, dequantizes AWQ/GPTQ on CPU (with HF→Meta RoPE rearrange on q_proj/k_proj), requants to Marlin layout in parallel, then installs on the model graph (main thread) with serial H2D copies.

Method Summary

Modifier and Type	Method	Description
static int	groupSizeFromConfig(Path checkpointDir)	Reads group_size from config.json, defaulting to 128.
static void	installLlamaLinears(LlamaModel model, Path dir, QuantFormat format, WeightGemmBackend backend, Device device, int groupSize, int tpSize, int tpRank, ScalarType outDtype)	Installs quantized projections for every Llama block.
static void	installLlamaLinears(LlamaModel model, Path dir, QuantFormat format, WeightGemmBackend backend, Device device, int groupSize, int tpSize, int tpRank, ScalarType outDtype, int modelLoaderThreads)	Installs quantized projections for every Llama block.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

installLlamaLinears

public static void installLlamaLinears(LlamaModel model,
 Path dir,
 QuantFormat format,
 WeightGemmBackend backend,
 Device device,
 int groupSize,
 int tpSize,
 int tpRank,
 ScalarType outDtype)
                                throws IOException

Installs quantized projections for every Llama block.

Parameters:

model - target model.

dir - checkpoint directory.

format - detected quantization format.

backend - GEMM backend to install.

device - target device.

groupSize - quantization group size.

tpSize - tensor-parallel world size.

tpRank - this rank.

outDtype - compute dtype for FP8 / NVFP4 output.

Throws:

IOException - if checkpoint IO fails.

See Also:

• installLlamaLinears(LlamaModel, Path, QuantFormat, WeightGemmBackend, Device, int, int, int, ScalarType, int)

installLlamaLinears

public static void installLlamaLinears(LlamaModel model,
 Path dir,
 QuantFormat format,
 WeightGemmBackend backend,
 Device device,
 int groupSize,
 int tpSize,
 int tpRank,
 ScalarType outDtype,
 int modelLoaderThreads)
                                throws IOException

Installs quantized projections for every Llama block.

Parameters:

model - target model.

dir - checkpoint directory.

format - detected quantization format.

backend - GEMM backend to install.

device - target device.

groupSize - quantization group size.

tpSize - tensor-parallel world size.

tpRank - this rank.

outDtype - compute dtype for FP8 / NVFP4 output.

modelLoaderThreads - pack/read concurrency hint (0 = auto).

Throws:

IOException - if checkpoint IO fails.

groupSizeFromConfig

public static int groupSizeFromConfig(Path checkpointDir)
                               throws IOException

Reads group_size from config.json, defaulting to 128.

Parameters:

checkpointDir - model directory.

Returns:

group size.

Throws:

IOException - if config IO fails.