Ant Group and vLLM open-source Humming, a JIT quantized-inference kernel library that supports T-Head's Zhenwu 810E and reports up to 2-5x Marlin speed on Hopper and Blackwell

Why it matters

【Key points】

Ant Group and the vLLM community have open-sourced Humming, a JIT-compiled GEMM kernel library for quantized inference, under Apache 2.0, announced on the official vLLM blog.

1. Approach: it separates the quantization description from execution choices and composes kernels from shared parts, instead of one kernel per weight/activation combination. Dense and MoE; WnA16, W8A8, W4A8, W4A4 with INT4, MXFP4, NVFP4 and more.

2. Hardware: every NVIDIA architecture from Turing on, plus CUDA-compatible chips such as T-Head's Zhenwu 810E.

3. Community microbenchmarks: dense W4A16 matches or beats vLLM's Marlin in most configs, ~2x on large-batch H200 and over 3x in some B300 cases; MoE MXFP4 ~2x on H20/H200, up to ~5x on B300's grouped path. FlashInfer and DeepGEMM are also baselines.

4. It uses two Blackwell tensor-core formats not exposed in PTX (float4e0m3, float8e3m4); Rubin lacks them.

【Verifiable】

Blog: https://vllm.ai/blog/2026-10-21-humming

Code: https://github.com/vllm-project/humming

【Our read】

1. Quantization is the main lever on inference cost, but formats keep multiplying and frameworks maintain separate kernels (Marlin, FlashInfer, DeepGEMM). Humming tries to unify them under one JIT framework.

2. It lands as DeepSeek open-sources its DeepInfer stack and AWS upstreams KV transfer to vLLM: big players are pushing inference internals into open frameworks. Ant's twist is first-class support for a Chinese chip, which matters to Chinese operators mixing NVIDIA and domestic hardware under export controls.

3. Caveats: kernel-level microbenchmarks, not end-to-end; no like-for-like numbers for the Zhenwu chip; the repo dates from February, and becoming vLLM's default path depends on later merges.