Reasoning Models, Stability Patches, and Local Inference Magic
Tools
Merged Apple Silicon Patches for llama.cpp
A practical fork for anyone tired of patching apple.py themselves. It merges the critical Metal kernel speedup and the prefill memory fix into a single build, making local development on Apple Silicon much more stable right now.
vLLM 0.8.5 Patches Disaggregated Serving Latency
Distributed serving just got a tangible stability boost here. It patches a messy issue with disaggregated serving where high latency could hang your request indefinitely, which is exactly the kind of edge case that kills your demo.
Keep an eye on this hub if you are deploying to Macs. They are keeping step with the official models but shipping them as ready-to-go 4-bit quantized imatrix files, saving you a weekend of messing with conversion scripts.
News
Microsoft Launches Phi-4-Reasoning-Plus
Microsoft is leaning hard into the 'reasoning' trend with this smaller Phi model. If you need a compact model that can actually plan step-by-step rather than just predict the next token, this is the current state of the art.
Analysis
Async Refactor in vLLM for Reliability
They solved the infamous hanging generation issues by refactoring the generation loop out of the engine. It is a lot of code, but it fixes the biggest reliability complaints we have had about multi-step inference.
Stay Ahead
Delivered each morning.