Package smile.llm.quant

package smile.llm.quant
Weight quantization backends: FP8 / NVFP4 (primary) and Marlin (Ampere failover).
  • Class
    Description
    DenseLinearRelease
    Releases placeholder dense LinearLayer shells after quantized ops are installed, so empty GPU weight storage does not inflate the KV budget.
    Fp8BlockDequant
    Dequantizes DeepSeek / Qwen-style fine-grained FP8 weights (weight_block_size = [128, 128] + weight_scale_inv).
    Fp8BlockLinear
    Block-scaled FP8 linear for DeepSeek / Qwen fine-grained FP8 checkpoints (weight_block_size=[128,128] + weight_scale_inv).
    Fp8KvCodec
    Helpers for FP8 KV store/load with per-tensor scales.
    Fp8Linear
    Weight-only FP8 linear layer using cuBLASLt via LibTorch _scaled_mm.
    LinearOp
    Shared forward interface for dense, FP8, NVFP4, and Marlin linear layers.
    MarlinLinear
    Marlin FP16×INT4 linear layer (Ampere/Ada failover for GPTQ/AWQ).
    MarlinWeightPacker
    Packs HuggingFace GPTQ / AWQ INT4 tensors into Marlin layout for MarlinLinear.
    MarlinWeightPacker.Packed
    Result of packing: Marlin qweight, scales, logical features.
    Nvfp4Linear
    NVFP4 weight-only linear for Blackwell (sm_100+).
    QuantBackendOverride
    Thread-local override for smile.chat.quantization during model load.
    QuantFormat
    Detected checkpoint quantization format.
    QuantFormatDetector
    Detects HuggingFace quantization format from config.json.
    QuantizedHfLoader
    Loads HuggingFace GPTQ / AWQ / FP8 / NVFP4 SafeTensors into Llama linears (shard-then-pack for TP when tpSize > 1).
    QuantizedQwenFp8Loader
    Installs native FP8 linears into a Qwen hybrid model from HuggingFace SafeTensors.
    QuantLinearFactory
    Factory for LinearOp backends from checkpoint tensors + selected backend.
    QuantPolicy
    Resolves checkpoint QuantFormat and WeightGemmBackend for load.
    QuantPolicy.Resolved
    Resolved quant policy for a checkpoint load.
    QuantTpSharding
    Shard-then-pack helpers for tensor-parallel quantized linears.
    WeightGemmBackend
    Weight GEMM backend selected from checkpoint format and GPU capability.