Local AI Digest: The Bare Metal Breakthrough

Tools

vLLM v0.6.0: Bare Metal Performance and New Scheduling

vLLM's latest release is a big deal for bare-metal builders. The new scheduler and quantization support mean you can squeeze way more tokens per second out of your existing hardware, which is the whole point of local.

Ollama: Tool Calling for Small Models and Faster Imports

Ollama just added native tool-calling support for models like Phi-3 and Gemma 2. This is huge because it brings agent-style capabilities to models that can actually run on a laptop, not just in the cloud.

MLC LLM: Better iOS Support and Leaner Builds

MLC LLM is making on-device inference more practical. Their latest update improves iOS performance and reduces binary size, which is critical if you're building an app that needs to ship without a backend.

Llamafile: Pushing the Single-File Frontier

Llamafile's single-binary approach keeps getting better. Their latest release improves support for more model architectures and adds better error handling, making it the ultimate 'just run it' tool for demos and quick tests.

LM Studio: The GUI Builder's Swiss Army Knife

LM Studio remains the best GUI for quickly testing and comparing models locally. Its latest update adds better model discovery and management, which saves a ton of time when you're evaluating what to actually use.

News

Phi-3 Mini: The Tiny Model That Keeps On Giving

Microsoft's Phi-3 Mini continues to dominate the 'small but capable' category. New fine-tunes and quantizations are popping up daily, proving it's a go-to foundation for local prototyping and edge deployment.