Ferrox

A pure-Rust, llama.cpp-free local LLM inference engine — smoke-tested end-to-end on a 3090/CUDA box in under ten minutes.

Goals

Test-drive antonellof/ferrox as a possible lighter-weight alternative to Ollama for local inference: build it against CUDA, run a real model through both the CLI and its OpenAI-compatible server, and get real numbers instead of trusting the README.

Effectiveness

Adequate, for what it actually demonstrated. It built clean, ran clean, and the docs matched real behavior on every step tested. But there's no evidence yet that it beats Ollama on this hardware specifically — ferrox's own published benchmarks are Metal/CPU-only, and CUDA support is explicitly flagged upstream as "kernels build and run on real hardware but have had no tuning pass." First look: positive. Verdict on actual value over Ollama: still open.

What made it effective

• cargo build --release --features cuda worked first try — CUDA 12.0 and the 3090 both auto-detected, no CPU fallback needed, ~39s build.
• CLI smoke test (TinyLlama-1.1B Q8_0, -dev cuda -ngl all) produced correct, coherent output at 62 t/s.
• ferrox-server's OpenAI-compatible /v1/chat/completions responded correctly with a clean start/stop — no config beyond pointing it at the GGUF.
• ferrox bench produced real numbers on the 3090 (pp512 = 50.6 t/s, tg128 = 73.7 t/s) that don't exist anywhere in the project's own published results — genuinely new data, not a re-run of someone else's numbers.

Bonus utility

The repo's own benchmarks/RESULTS.md has zero CUDA rows — Metal and CPU only. ferrox bench --suite run on a 3090 and upstreamed would be a small, legitimate contribution rather than a one-off local curiosity.

Friction, pain points, surprises

• The tool's headline differentiator — decode speed wins over llama.cpp — is measured entirely on Apple Silicon in the docs, and that gap isn't flagged loudly enough. Cost a round-trip here: the first answer to "what's better than Ollama?" leaned on those Metal numbers before catching that none of them transfer to a 3090/CUDA/Linux box. The fix in hindsight is boring but important — cross-check the user's actual documented hardware before repeating a repo's own performance claims, not just what the repo happens to publish.
• No Vulkan backend at all — compiled targets are CPU, Metal, CUDA only. Anyone routing through Vulkan for driver-compat reasons doesn't have that option.
• CUDA is explicitly unfinished on ferrox's own roadmap (kernels run, no tuning pass yet), so the honest comparison against Ollama's llama.cpp-CUDA path is still an open question, not a settled "yes."