The foundational C/C++ inference engine for running LLMs and VLMs locally, created by Georgi Gerganov in March 2023. It has no external dependencies, runs on everything from Raspberry Pi to NVIDIA GPUs, and defines the GGUF file format that has become the de facto standard for distributing quantized open models. Its core strengths are extreme resource efficiency (1.5-bit to 8-bit quantization lets 7B models run in ~4GB RAM), broad hardware support (CPU, Apple Metal, CUDA, Vulkan, SYCL, HIP, WebGPU), and fine-grained control over every inference parameter. It is the engine underneath Ollama and LM Studio. The project is MIT-licensed and actively maintained (v0.9.0 released November 2026). Installation is available via a one-line script (llama.app), Docker, pre-built binaries, or source build. It is not a GUI application—users interact via CLI (llama cli, llama serve) or its OpenAI-compatible server.
Pros & Cons:
Pros: The most efficient way to run LLMs on constrained hardware—quantization and C/C++ implementation mean models run on devices where other engines cannot, including Raspberry Pi and Android; it defines GGUF, which means the widest model availability (any GGUF on Hugging Face works) and compatibility with the broader ecosystem; fine-grained control over quantization, context size, GPU layer offload (-ngl), and every inference parameter allows optimization that wrapper tools hide; multi-backend support covers virtually every hardware target, from CPU-only to multi-GPU; MIT licensed and community-maintained with rapid iteration.
Cons: It is not user-friendly—there is no GUI, every action requires CLI flags, and getting the best performance often means compiling from source with the right CPU/GPU flags; model management is manual—you find and download GGUF files yourself, manage quantization choices, and handle model paths, unlike Ollama’s registry or LM Studio’s visual browser; documentation is scattered across the repo, wiki, and community discussions, making best practices harder to discover; performance tuning is non-trivial—choosing the right -ngl value, context size, and batch parameters requires understanding your hardware; it is an engine, not a product, so teams wanting managed deployment, multi-user serving, or production SLAs should look at vLLM or managed platforms.
Who is it for?:Developers deploying LLMs on edge devices, embedded systems, or hardware without dedicated GPUs; users who need full control over quantization and inference parameters for performance or memory optimization; anyone building a local inference stack who wants to understand or modify the underlying engine (Ollama and LM Studio users are indirectly using it); teams working with unusual model architectures or custom quantizations that wrapper tools don’t support; privacy-focused users who want a minimal, auditable codebase with no telemetry. Not suited for non-technical users seeking a GUI (LM Studio), developers wanting one-command model management (Ollama), or teams needing high-concurrency production serving (vLLM).