Local AI Digest: New Small Model Drops, GGUF Fixes, and the Agentic Runtime Race

Tools

llama.cpp Patches Long-Standing Quantization Rounding Bug

A subtle rounding issue in some quant kernels was quietly degrading output quality on certain hardware, and the fix landed with minimal fanfare. Worth re-pulling and re-quantizing your models if you've noticed weird degradation that benchmarks didn't explain.

vLLM Adds Better Multi-GPU Scheduling for Local Clusters

If you're running a small local cluster instead of a single beefy box, the new scheduling logic squeezes more throughput out of mismatched GPU setups. Practical win for anyone self-hosting inference on a mix of older and newer cards.

Analysis

Paper Argues Most Open Benchmarks Overstate Real-World Coding Ability

The authors show a decent chunk of coding benchmark gains come from memorized patterns rather than generalizable reasoning, which should make you skeptical of leaderboard flexing. Test on your own private eval set before you trust a benchmark score to pick your next local coding model.