Release v1.0.0 · ggml-org/ggml · GitHub

Overview

ggml 1.0.0 prunes the deprecated public API and the legacy DIAG_MASK ops, adds configurable YaRN truncation and FP8 support, and brings broad performance work across backends: few-row quantized matmuls and MoE fusion on CUDA/Metal, a Vulkan prefill FlashAttention path, a device-side MoE scheduler on SYCL, decomposed Adreno FlashAttention in OpenCL, new WebGPU kernels, expanded Hexagon op coverage, and scheduler improvements that avoid uploading unused inputs.

API changes

• Pruned deprecated API and removed the legacy DIAG_MASK_INF/DIAG_MASK_ZERO ops, bumping the RPC protocol to reject incompatible peers (llama/29801, llama/31946)
• Added ggml_rope_set_truncate/ggml_rope_get_truncate to control YaRN correction-bound rounding (llama/30206, llama/30802)
• Implemented the backward pass for TRI (llama/30712)
• Added MUL_MAT src1 precision tests and gating for ggml_prec_set_acc (llama/31896, llama/31851)
• Bumped the RPC protocol to v9 with faster loading and graceful failure on unreachable endpoints or failed handshakes (llama/29413, llama/30415)

Core changes

• Scheduler skips uploading inputs that are not read and prefetches only inputs produced by earlier splits (llama/31419, llama/32100)
• MoE cache can compute a share of the cache misses on the CPU and copy the results back asynchronously (llama/30966)
• Meta backend raises the compute view limit for buffers with few tensors (llama/31447)

Backend changes

CUDA

• Added few-row MMQ for batch sizes 2-24 and MMVF for BF16 few-row matmuls (llama/31533, llama/31959)
• Added OCP FP8 support and fused NVFP4 gate/up matmuls with GLU in MMQ (llama/28413, llama/31296)
• Extended FlashAttention with an sm70 head-size-256 config and MMA head sizes 40/72 (llama/27997, llama/30285)
• Improved graph handling: skip compat checks on reuse, re-instantiate on cluster-dim changes, move graph state (llama/32004, llama/31284, llama/32001)
• MoE: skip empty MMQ stream-k tiles, linearize the MoE layout, and sanitize post-bias NaNs (llama/32067, llama/31047, llama/31005)
• Added a debug buffer pool and halved the VMM pool reservation with reserve retries (llama/31919, llama/30542)
• HIP: use a fixed BLAS workspace and add an MFMA lightning indexer for CDNA2 (llama/31838, llama/29050)
• Added a d_state=64 SSM_SCAN fast path and made radix Top-K deterministic for ties (llama/31818, llama/31084)

Metal

Vulkan

• Added an Intel FlashAttention prefill kernel (llama/29357)
• Added batched tensor copies, a test-fusion interface, and ReBAR detection (llama/31418, llama/30769, llama/31658)
• Enabled dequantized KV with CM2, YaRN truncation, RDNA3/4 large-tile split-K, and large coopmat matmul on newer AMD drivers (llama/31526, llama/30802, llama/30280, llama/31587)
• Added a tiled-transpose concat shader and FWHT kernels for block widths above 512 (llama/30591, llama/29772)
• Fixed MoltenVK out-of-bounds accesses, GROUP_NORM workgroup limits, and FA scratch sizing (llama/31890, llama/31930, llama/30487)
• Capped scalar FlashAttention at 8 rows for large heads on Qualcomm (llama/30829)

SYCL

OpenCL

• Added decomposed Adreno FlashAttention and improved decode performance (llama/32046, llama/31633)
• Added q6_k tiled, q8_0, and f32/q5_k multi-column GEMV kernels and f32 MUL_MAT_ID (llama/31758, llama/30947, llama/31783)
• Added arange, non-contiguous sigmoid, repeat for f16/i32/i16/bf16, and DSV4 HC pre/post kernels (llama/31829, llama/31687, llama/31782, llama/31623)
• Enabled multi-column GEMV by default for Adreno X2 (llama/31694)

WebGPU

Hexagon

CPU

CANN

• Fail early when a buffer type is unsupported (llama/31270)

ZDNN

• Accept non-zdnn tensors, invalidate compute tensors after each op, and fix release backend lib detection (llama/31824, llama/31905)

OpenVINO

More info

Changelog since v0.29.0

417a024 ggml : bump version to 1.0.0 (#1690)
771b3d9 requirements : fix numpy for python 3.14 (#1691)
f154959 sync : llama.cpp
ec04faf fix(hexagon): make host build work with clang-cl on Windows ARM64 (llama/32011)
b45738e examples : adapt to removal of ggml_diag_mask_inf
1514439 sync : llama.cpp
5f29555 ggml : prune (llama/29801)
076861c hexagon: add Q5_0 and Q5_1 support (llama/32094)
63ec52d Title: CUDA: few-row MMQ for BS 2-24 (llama/31533)
c6fe796 ggml : route GATED_DELTA_NET to the SpacemiT RVV kernel (llama/31897)
4ae6e72 vulkan : Intel FA kernel for prefill. (llama/29357)
b22071c ggml-cpu: use the 4×4 simd_gemm tile on ARM NEON (llama/32128)
ba8b013 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (llama/26048)
05f605a ggml-backend : prefetch only inputs computed by earlier splits (llama/32100)
b8a5b87 ggml : dispatch SpacemiT Q4_1 32×256 through the HP kernel (llama/31929)
7a8cef5 sycl: make oneDNN FA cache thread-safe and memoize runtime failures (llama/31138)
6dfea1a sycl : fix AOT for more than 4GB (llama/31873)
012ff44 sycl : use cached device-id in set_main_device (llama/32013)
2f9d47b CUDA: add cuda debug buffer pool (llama/31919)
522d257 opencl: add bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin (llama/30947)
0f9dcb7 CUDA: skip empty MoE tiles in MMQ stream-k fixup (llama/32067)
6bed9a8 opencl: add decomposed FA for Adreno (llama/32046)
a357e7b metal : prevent out-of-bounds reads in SOLVE_TRI kernel (llama/32103)
c36ccd8 vulkan: support batched tensor copy API (llama/31418)
1bffc59 backend : re-instantiate CUDA graphs when the cluster dims of a kernel change (llama/31284)
8153155 ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call (llama/31838)
ad7d200 cpu: accumulate flash attention V in F32 (llama/29810)
7f4a2db webgpu: add PAD_REFLECT_1D support (llama/31877)
0f51454 vulkan: allow dequantized KV optimization with CM2 (llama/31526)
eb4de49 vulkan: allocate the device fault vendor binary buffer (llama/32008)
6b7f47a cuda : use MMVF for BF16 matrices of few rows at small batch sizes (llama/31959)
77d01f1 cuda : add sm70 FlashAttention config case for DKQ=256, DV=256, ncols=64 (llama/27997)
bd37ba0 cuda : fattn mma head sizes 40 72 (llama/30285)
161ca15 CUDA: Add OCP FP8 support (llama/28413)
06faf80 sycl : fixed sycl CLAMP to actually support strided view (llama/31784)
a986181 cuda : remove unused ids and type_src0 parameters (llama/32018)
55104b2 llama : Add FP8 KV-cache support (llama/28392)
0753641 ggml : add MUL_MAT src1 precision tests and gating (llama/31896)
1cd16bd cuda : remove unused quantize_cuda_t typedef (llama/31884)
4a18cdd cuda : halve VMM pool reservation size and retry on cuMemAddressReserve failure (llama/30542)
1c9ac9d CUDA: skip graph compat check when graph is being reused (llama/32004)
999c725 ggml : fix FreeBSD Vulkan build by setting _XOPEN_SOURCE=700 (llama/31701)
47bc1d8 CUDA: move graph and concurrent stream state out of common.cuh (llama/32001)
df70e12 cann: fail early when checking if buffer type is supported (llama/31270)
e447eb5 sycl: Add device-side MoE scheduler (llama/30156)
9a1c582 vulkan: add tiled-transpose concat shader for Qwen models (llama/30591)
8b3c8cf ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (llama/30415)
f860f1d vulkan: Divide GROUP_NORM workgroup to 2D to avoid maxComputeWorkGroupCount limit (llama/31930)
eb5571a vulkan: Fix out of bounds reads/writes on MoltenVK (llama/31890)
5a9a264 vulkan: cap scalar flash attention at 8 rows for large heads on Qualcomm (llama/30829)
5610f41 sycl: un-gate oneDNN fused SDPA for Alchemist at head_dim=64 (llama/31574)
487cadf hexagon: fix 64-bit FastRPC handle truncation on close (llama/30952)
be37c9c ggml-zdnn: accept non-zdnn tensors, invalidate compute tensors after every op (llama/31824)
294d5fc ggml : implement the backward pass for TRI (llama/30712)
e18c2c6 ggml-backend: don’t upload inputs that are not read (llama/31419)
e84d36b metal : prevent out-of-bounds writes to the Metal FA output buffer (llama/31662)
0dd184f metal : make ggml_tri select instead of scaling by the comparison (llama/30711)
b6d482c opencl : include climits (llama/31935)
62de780 RPC: Improve loading times (llama/29413)
a7276d0 ggml : support YaRN truncation config on openvino, sycl, vulkan, webgpu, rpc (llama/30802)
da89459 sycl: cache device-id (llama/31126)
fb57085 tests: bound the test_rope tolerance by the f32 angle resolution (llama/31874)
e43d847 opencl: add f32 MUL_MAT_ID (llama/31783)
d1c1e4b ggml-zdnn: fix release binary not detecting backend lib (llama/31905)
7e7b6e9 mtmd: refactoring: remove clip.h, consolidate code into mtmd (llama/31865)
5a0e536 cpu: enable SpacemiT IME2 MXFP4 dense repack (llama/31776)
7f4b25f llamafile: impl bf16 and q8_0 vxe (llama/31866)
f22fc6a ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (llama/31084)
47b8b4f ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (llama/31818)
b524960 webgpu : add COUNT_EQUAL support (llama/31578)
75a4f5a ggml : switch to F32 src1 in the CPU backend (llama/31854)
6cddab2 llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (llama/31851)
7353275 opencl: improve q8_0 and MoE small-n verification for spec (llama/31831)
8f1404e hexagon: support for automatic loading of libhtp without legacy env vars (llama/31781)
91c102a sycl: exact rounding in CPY to Q4_1, Q5_1 and Q8_0 (llama/31465)
8cf32bc sycl: refuse device buffers that would exceed VRAM (llama/31285)
d969b07 hexagon: n_hvx and n_threads cleanup (llama/31857)
5746ebd llama_moe_cache: run missed experts on CPU, add tuning script (llama/30966)
a49f174 sycl: add f16 support for SWIGLU_OAI (llama/30857)
c6a0780 metal : fix GROUP_NORM for ne03 > 1 and non-contiguous inputs (llama/31852)
1e5c3e0 webgpu : Implement group norm kernel (llama/31490)
37ecdca vulkan: Add ‘readonly’ to decodeBuf declarations. (llama/31833)
c08538b CUDA: fix shared memory bank conflicts in the lightning indexer (llama/31803)
07b8362 opencl: add arange (llama/31829)
f55a504 meta : raise the compute view limit for buffers with few tensors (llama/31447)
2870133 hexagon: fuse post-norm residual add into RMS_NORM_MUL (llama/31822)
a135ec6 hexagon: conv_2d optimizations and support for conv_3d (llama/31814)
e3d3633 tests : qualify isnan and signbit with std:: (llama/31791)
a7d3330 tests : override hardware_concurrency in CI (llama/31797)
ab7ca9c opencl: support repeat for f16, i32, i16, bf16 (llama/31782)
9290392 webgpu: add batched tensor API (get_tensor_n and set_tensor_n) (llama/31473)
f594fed vulkan: detect ReBAR support, avoid BAR allocations if unavailable (llama/31658)
38e4be2 CUDA: fix Maxwell build (llama/31787)
8bd6b61 opencl: add q6_k multi column tiled GEMV for speculative verification and refactoring (llama/31758)
d3ce63d hexagon: Implement set_tensor_n and improve repacking/tiling robustness (llama/31744)
9b1880f ggml-cuda: sanitize post-bias NaNs in MoE selection (llama/31005)
7e607b8 vulkan: Implement test-fusion interface (llama/30769)
2188d13 CUDA: use highest compiled arch in get_mmvq_mmid_max_batch (llama/31679)
114ce5e vulkan: RDNA3/4 Large-Tile Split-K Optimization (llama/30280)
e074347 metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (llama/31615)
f0f5b42 CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a) (llama/29050)
fa8c617 CUDA: linearize MoE data layout for MMQ/MMF (llama/31047)
1690d2e vulkan: FWHT kernels for block widths above 512 (llama/29772)
40278c4 vulkan : check combined K+V f16 scratch size in FA dequant path (llama/30487)
2dccfdb sync : llama.cpp
25b845e ggml : support truncate=false for YaRN (llama/30206)
accf2a1 sync : llama.cpp
b2ce926 metal : use mul_mm for 32 <= K < 64 (llama/30982)
1a5f477 webgpu : add MEAN support (llama/31591)
2ab6d6c vulkan: Re-enable large coopmat matmul configuration on AMD proprietary driver 26.10.1+ (llama/31587)
19df6cf tests: fix CI flakes of TOPK_MOE caused by near ties (llama/31426)
a0df99d metal: keep fusable node groups inside one command buffer (llama/31688)
3a996db opencl: enable f32 and q5_k multi column GEMV by default for Adreno X2 (llama/31694)
d75184e opencl: support non-contig sigmoid (llama/31687)
c634501 opencl: add DSV4_HC_PRE and DSV4_HC_POST (llama/31623)
0816137 Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (llama/31296)
4ac0dae opencl: improve decode perf with FA (llama/31633)
4f681f2 webgpu: return maxBufferSize from get_max_size and avoid whole-buffer bindings in memset/clear (llama/31348)
cd6d2b7 docs : sync ggml_type enum in gguf.md (#1685)