Class MarlinWeightPacker

java.lang.Object
smile.llm.quant.MarlinWeightPacker
public final class MarlinWeightPacker extends Object
Packs HuggingFace GPTQ / AWQ INT4 tensors into Marlin layout for MarlinLinear. Invoked only when WeightGemmBackend.MARLIN is selected (Ampere/Ada).

Default HF load path uses packAwqDirect(Tensor, Tensor, Tensor, int, Device, Integer) / packGptqDirect(Tensor, Tensor, Tensor, Tensor, int, Device, Integer) (dequant → optional RoPE permute → Marlin requant). packFromFp16(Tensor, int, Device) remains for tests and dense requant.

  • Method Details

    • packGptq

      public static MarlinWeightPacker.Packed packGptq(Tensor qweight, Tensor scales, Tensor qzeros, Tensor gIdx, int groupSize, Device device)
      Packs GPTQ tensors into Marlin layout (FP16 dequant→requant; act-order rejected).
      Parameters:
      qweight - GPTQ packed weights.
      scales - group scales.
      qzeros - packed zeros, or null.
      gIdx - GPTQ g_idx, or null.
      groupSize - quantization group size.
      device - destination device.
      Returns:
      Marlin-packed tensors.

    • packAwq

      public static MarlinWeightPacker.Packed packAwq(Tensor qweight, Tensor scales, Tensor qzeros, int groupSize, Device device)
      Packs AutoAWQ GEMM tensors into Marlin layout.

      Dequantizes with AWQ zeros (exact), then requants into Marlin's symmetric int4 (zp=8). Folding zeros into int4 with clamp is unsafe when |q - z| > 7.

      Parameters:
      qweight - AWQ packed weights.
      scales - group scales.
      qzeros - packed zeros, or null.
      groupSize - quantization group size.
      device - destination device.
      Returns:
      Marlin-packed tensors.

    • packAwqDirect

      public static MarlinWeightPacker.Packed packAwqDirect(Tensor qweight, Tensor scales, Tensor qzeros, int groupSize, Device device, Integer ropeHeads)
      AWQ→Marlin pack with optional HF→Meta RoPE rearrange on out-features.
      Parameters:
      qweight - AWQ packed weights.
      scales - group scales.
      qzeros - packed zeros, or null.
      groupSize - quantization group size.
      device - destination device.
      ropeHeads - if non-null, permute [out,in] for Meta-style RoPE (num_attention_heads or num_kv_heads).
      Returns:
      Marlin-packed tensors.

    • packAwqDirect

      public static MarlinWeightPacker.Packed packAwqDirect(Tensor qweight, Tensor scales, Tensor qzeros, int groupSize, Device device)
      AWQ→Marlin pack without RoPE rearrange.
      Parameters:
      qweight - AWQ packed weights.
      scales - group scales.
      qzeros - packed zeros, or null.
      groupSize - quantization group size.
      device - destination device.
      Returns:
      Marlin-packed tensors.
      See Also:

    • reversePermuteHfToMeta

      public static Tensor reversePermuteHfToMeta(Tensor w, int nHeads)
      HuggingFace interleaved Q/K layout → Meta / GPT-NeoX layout used by Smile RoPE. Same transform as dense Llama.reversePermute on [out, in] weights.
      Parameters:
      w - weight [out, in].
      nHeads - attention or KV head count.
      Returns:
      permuted weight (caller owns).

    • packGptqDirect

      public static MarlinWeightPacker.Packed packGptqDirect(Tensor qweight, Tensor scales, Tensor qzeros, Tensor gIdx, int groupSize, Device device, Integer ropeHeads)
      Direct GPTQ→Marlin pack. Act-order (g_idx not sequential) fails fast.

      Dequantizes (with zeros when present), optionally applies HF→Meta RoPE rearrange, then requants into Marlin symmetric int4.

      Parameters:
      qweight - GPTQ packed weights.
      scales - group scales.
      qzeros - packed zeros, or null.
      gIdx - GPTQ g_idx, or null.
      groupSize - quantization group size.
      device - destination device.
      ropeHeads - if non-null, permute out-features for Meta-style RoPE.
      Returns:
      Marlin-packed tensors.

    • packGptqDirect

      public static MarlinWeightPacker.Packed packGptqDirect(Tensor qweight, Tensor scales, Tensor qzeros, Tensor gIdx, int groupSize, Device device)
      Direct GPTQ→Marlin pack without RoPE rearrange.
      Parameters:
      qweight - GPTQ packed weights.
      scales - group scales.
      qzeros - packed zeros, or null.
      gIdx - GPTQ g_idx, or null.
      groupSize - quantization group size.
      device - destination device.
      Returns:
      Marlin-packed tensors.
      See Also:

    • packFromFp16

      public static MarlinWeightPacker.Packed packFromFp16(Tensor weightFp16, int groupSize, Device device)
      Packs a dense FP16 weight [out, in] into Marlin INT4 layout matching upstream marlin.Layer.pack (B is [k/16, n*16/8]).
      Parameters:
      weightFp16 - dense FP16 weight [out, in].
      groupSize - quantization group size.
      device - destination device.
      Returns:
      Marlin-packed tensors.