llama.cpp

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.

When to use llama.cpp

Use llama.cpp when:

Running on CPU-only machines Deploying on Apple Silicon (M1/M2/M3/M4) Using AMD or Intel GPUs (no CUDA) Edge deployment (Raspberry Pi, embedded systems) Need simple deployment without Docker/Python

Use TensorRT-LLM instead when:

Have NVIDIA GPUs (A100/H100) Need maximum throughput (100K+ tok/s) Running in datacenter with CUDA

Use vLLM instead when:

Have NVIDIA GPUs Need Python-first API Want PagedAttention Quick start Installation

macOS/Linux

brew install llama.cpp

Or build from source

git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make

With Metal (Apple Silicon)

make LLAMA_METAL=1

With CUDA (NVIDIA)

make LLAMA_CUDA=1

With ROCm (AMD)

make LLAMA_HIP=1

Download model

Download from HuggingFace (GGUF format)

huggingface-cli download \ TheBloke/Llama-2-7B-Chat-GGUF \ llama-2-7b-chat.Q4_K_M.gguf \ --local-dir models/

Or convert from HuggingFace

python convert_hf_to_gguf.py models/llama-2-7b-chat/

Run inference

Simple chat

./llama-cli \ -m models/llama-2-7b-chat.Q4_K_M.gguf \ -p "Explain quantum computing" \ -n 256 # Max tokens

Interactive chat

./llama-cli \ -m models/llama-2-7b-chat.Q4_K_M.gguf \ --interactive

Server mode

Start OpenAI-compatible server

./llama-server \ -m models/llama-2-7b-chat.Q4_K_M.gguf \ --host 0.0.0.0 \ --port 8080 \ -ngl 32 # Offload 32 layers to GPU

Client request

curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama-2-7b-chat", "messages": [{"role": "user", "content": "Hello!"}], "temperature": 0.7, "max_tokens": 100 }'

Quantization formats GGUF format overview Format Bits Size (7B) Speed Quality Use Case Q4_K_M 4.5 4.1 GB Fast Good Recommended default Q4_K_S 4.3 3.9 GB Faster Lower Speed critical Q5_K_M 5.5 4.8 GB Medium Better Quality critical Q6_K 6.5 5.5 GB Slower Best Maximum quality Q8_0 8.0 7.0 GB Slow Excellent Minimal degradation Q2_K 2.5 2.7 GB Fastest Poor Testing only Choosing quantization

General use (balanced)

Q4_K_M # 4-bit, medium quality

Maximum speed (more degradation)

Q2_K or Q3_K_M

Maximum quality (slower)

Q6_K or Q8_0

Very large models (70B, 405B)

Q3_K_M or Q4_K_S # Lower bits to fit in memory

Hardware acceleration Apple Silicon (Metal)

Build with Metal

make LLAMA_METAL=1

Run with GPU acceleration (automatic)

./llama-cli -m model.gguf -ngl 999 # Offload all layers

Performance: M3 Max 40-60 tokens/sec (Llama 2-7B Q4_K_M)

NVIDIA GPUs (CUDA)

Build with CUDA

make LLAMA_CUDA=1

Offload layers to GPU

./llama-cli -m model.gguf -ngl 35 # Offload 35/40 layers

Hybrid CPU+GPU for large models

./llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20 # GPU: 20 layers, CPU: rest

AMD GPUs (ROCm)

Build with ROCm

make LLAMA_HIP=1

Run with AMD GPU

./llama-cli -m model.gguf -ngl 999

Common patterns Batch processing

Process multiple prompts from file

cat prompts.txt | ./llama-cli \ -m model.gguf \ --batch-size 512 \ -n 100

Constrained generation

JSON output with grammar

./llama-cli \ -m model.gguf \ -p "Generate a person: " \ --grammar-file grammars/json.gbnf

Outputs valid JSON only

Context size

Increase context (default 512)

./llama-cli \ -m model.gguf \ -c 4096 # 4K context window

Very long context (if model supports)

./llama-cli -m model.gguf -c 32768 # 32K context

Performance benchmarks CPU performance (Llama 2-7B Q4_K_M) CPU Threads Speed Cost Apple M3 Max 16 50 tok/s $0 (local) AMD Ryzen 9 7950X 32 35 tok/s $0.50/hour Intel i9-13900K 32 30 tok/s $0.40/hour AWS c7i.16xlarge 64 40 tok/s $2.88/hour GPU acceleration (Llama 2-7B Q4_K_M) GPU Speed vs CPU Cost NVIDIA RTX 4090 120 tok/s 3-4× $0 (local) NVIDIA A10 80 tok/s 2-3× $1.00/hour AMD MI250 70 tok/s 2× $2.00/hour Apple M3 Max (Metal) 50 tok/s ~Same $0 (local) Supported models

LLaMA family:

Llama 2 (7B, 13B, 70B) Llama 3 (8B, 70B, 405B) Code Llama

Mistral family:

Mistral 7B Mixtral 8x7B, 8x22B

Other:

Falcon, BLOOM, GPT-J Phi-3, Gemma, Qwen LLaVA (vision), Whisper (audio)

Find models: https://huggingface.co/models?library=gguf

References Quantization Guide - GGUF formats, conversion, quality comparison Server Deployment - API endpoints, Docker, monitoring Optimization - Performance tuning, hybrid CPU+GPU Resources GitHub: https://github.com/ggerganov/llama.cpp Models: https://huggingface.co/models?library=gguf Discord: https://discord.gg/llama-cpp