vLLM Drops Massive Update, Spec Decoding for VLMs, and DeepSeek Gets a Brain Boost

Tools

vLLM Disaggregated Inference Drops

The vLLM inference engine gets a prefill-decode disaggregation upgrade, which is critical for serving massive models locally or in a distributed setup. This lets you scale prefill and decode independently, making high-throughput local inference actually practical.

Speculative Decoding for VLMs in llama.cpp

A pending llama.cpp PR adds speculative decoding support for vision-language models. This is a big deal for local inference: VLMs are heavy, and speculative decoding can give a meaningful speed boost on consumer GPUs.

Models

DeepSeek-V3-0324 Model Card Update

DeepSeek's 0324 checkpoint improves reasoning and coding performance and is now readily available for download and fine-tuning. This is one of the strongest open-weight models you can run on your own hardware.

SmolLM3 Model Gets Maintenance Update

The SmolLM3 model from HuggingFace keeps getting brushed up, with a focus on tool use and a small footprint. This is a great example of a "tactical" model tuned for specific, constrained use cases rather than general benchmarks.

Analysis

AutoSearch: Automated RAG Optimization

The arxiv paper on AutoSearch discusses an auto-search framework for retrieval augmented generation. The key insight is automating RAG pipeline tuning. For builders, this is about less manual prompt engineering and more systematic optimization.