llama.cpp Quickstart with CLI and Server

llama.cpp is a C/C++ inference engine for GGUF models: llama-cli for interactive chat, llama-completion for scripted prompts, and llama-server for an OpenAI-compatible HTTP API.

If you are still deciding between local, self-hosted, and cloud approaches, start with the pillar guide LLM Hosting in 2026: Local, Self-Hosted & Cloud Infrastructure Compared.

Why llama.cpp in 2026

llama.cpp is a lightweight inference engine with a bias toward:

• portability across CPUs and multiple GPU backends,
• predictable latency on a single machine,
• deployment flexibility, from laptops to on-prem nodes.

It shines when you want privacy and offline operation, when you need deterministic control over runtime flags, or when you want to embed inference into a larger system without running a full Python-heavy stack.

It is also helpful to understand llama.cpp even if you later choose a higher-throughput server runtime. For example, if your goal is maximum serving throughput on GPUs, you might also want to compare it to vLLM using: vLLM Quickstart: High-Performance LLM Serving and you can benchmark tool choices in: Ollama vs vLLM vs LM Studio: Best Way to Run LLMs Locally in 2026?.

If Ollama specifically is the alternative you are weighing against llama-server, llama.cpp vs Ollama in 2026 is the dedicated pairwise comparison, with concrete triggers for when to keep Ollama and when to move to direct llama.cpp.

Install llama.cpp on Windows, macOS, and Linux

Commands below match llama.cpp v0.6.0 (5 October 2026). A catalog entry for the same runtime — GGUF inference on CPU and accelerator backends, versioned by build commit — is at the llama.cpp runtime reference on LLM Systems. There are four practical install paths, depending on whether you want convenience, portability, or maximum performance.

Install via package managers

This is the fastest “get it running” option.

# macOS or Linux
brew install llama.cpp

# Windows
winget install llama.cpp

# macOS (MacPorts)
sudo port install llama.cpp

# macOS or Linux (Nix)
nix profile install nixpkgs#llama-cpp

# Ubuntu (universe). This package lags upstream by a long way:
# 8681+dfsg against current build b11433 / v0.6.0, so newer flags will be missing.
sudo apt install llama.cpp-tools

Tip: after installing, verify the tools exist:

llama-cli --version
llama-completion --version
llama-server --version

Install via pre-built binaries

If you want a clean install without compilers, use the official pre-built binaries published in the llama.cpp GitHub releases. They typically cover multiple OS targets and multiple backends (CPU-only and GPU-enabled variants).

A common workflow:

# 1) Download the right archive for your OS and backend
# 2) Extract it
# 3) Run from the extracted folder

./llama-cli --help
./llama-server --help

Run the official container

The project publishes server images on GitHub Container Registry. The CPU image is ghcr.io/ggml-org/llama.cpp:server; the CUDA image is :server-cuda. The container must listen on all interfaces, because the default bind is still 127.0.0.1:

docker run --gpus all -p 8080:8080 \
  -v /path/to/models:/models \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m models/my-model.gguf -c 4096 --host 0.0.0.0 -ngl all

Build from source for your exact hardware

If you care about squeezing the best performance out of your CPU/GPU backend, build from source with CMake.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# CPU build
cmake -B build
cmake --build build --config Release

After build, binaries are typically here:

ls -la ./build/bin/

GPU builds in one command

Enable the backend that matches your hardware (examples shown for CUDA and Vulkan):

# NVIDIA CUDA
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Vulkan
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Ubuntu 24.04 + NVIDIA GPU: full build walkthrough

On Ubuntu 24.04 with an NVIDIA GPU, you need the CUDA toolkit and OpenSSL before building. Here is a tested sequence:

1. Install CUDA toolkit 13.1. This walkthrough pins the 13.1.1 local installer. A newer toolkit from NVIDIA’s Ubuntu 24.04 repository follows the same pin-file and local-deb steps with updated filenames.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
sudo mv cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/13.1.1/local_installers/cuda-repo-ubuntu2404-13-1-local_13.1.1-590.48.01-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2404-13-1-local_13.1.1-590.48.01-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2404-13-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-13-1

2. Add CUDA to your environment (append to ~/.bashrc):

# cuda toolkit
export PATH=/usr/local/cuda-13.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-13.1/lib64:$LD_LIBRARY_PATH

Then run source ~/.bashrc or open a new terminal.

3. Install OpenSSL development headers (required for a clean build):

sudo apt update
sudo apt install libssl-dev

4. Build llama.cpp (from the directory containing your llama.cpp clone, with CUDA enabled):

cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split llama-embedding
cp llama.cpp/build/bin/llama-* llama.cpp

This produces llama-cli, llama-mtmd-cli, llama-server, llama-embedding, and llama-gguf-split in the llama.cpp directory.

You can also compile multiple backends and choose devices at runtime. This is useful if you deploy the same build onto heterogeneous machines.

Pick a GGUF model and a quantization

To run inference, you need a GGUF model file (*.gguf). GGUF is a single-file format that bundles model weights plus standardized metadata needed by engines like llama.cpp.

Two ways to get a model

Option A: Use a local GGUF file

Download or copy a GGUF into ./models/:

mkdir -p models
# Place your GGUF at models/my-model.gguf

Then run it by path:

llama-cli -m models/my-model.gguf -p "Hello! Explain what llama.cpp is." -n 128 -st

Option B: Let llama.cpp download from Hugging Face

Modern llama.cpp builds can download from Hugging Face and keep files in a local cache. This is often the easiest workflow for quick experiments.

# Download a model from HF and run a prompt
llama-cli \
  --hf-repo ggml-org/tiny-llamas \
  --hf-file stories15M-q4_0.gguf \
  -p "Once upon a time," \
  -n 200 \
  -st

You can also specify the quant in the repo selector and let the tool select a matching file:

llama-cli \
  --hf-repo unsloth/phi-4-GGUF:q4_k_m \
  -p "Summarize the concept of quantization in one paragraph." \
  -n 160 \
  -st

If you need a fully offline workflow later, --offline forces cache usage and prevents network access.

Quantization choice for local inference

Quantization is the practical answer to the question “Which GGUF quantization should you choose for local inference” because it directly trades off quality, model size, and speed.

A pragmatic starting point:

• start with a Q4 or Q5 variant for CPU-first machines,
• move to higher precision (or less aggressive quantization) when you can afford the RAM or VRAM,
• when the model “feels dumb” for your task, the fix is often either a better model or a less aggressive quant, not only sampling tweaks.

Also remember context window matters: larger context sizes increase memory usage (sometimes dramatically), even when the GGUF file itself fits.

llama-cli quickstart and key parameters

llama-cli is the fastest way to validate that your model loads, your backend works, and your prompts behave.

llama-cli has been an interactive chat client since the December 2025 rewrite. When the model ships a chat template, conversation mode is on by default, and --no-conversation is rejected with a pointer to llama-completion. That second binary is the old llama-cli: one prompt in, one completion out, suitable for scripts and pipes. From the new client, --single-turn (-st) generates one reply and exits.

One reply, then exit

llama-cli \
  -m models/my-model.gguf \
  -p "Write a short TCP vs UDP comparison." \
  -n 200 \
  -st

The same prompt through the non-interactive binary:

llama-completion \
  -m models/my-model.gguf \
  -p "Write a short TCP vs UDP comparison." \
  -n 200 \
  --no-conversation

Interactive chat run

llama-cli \
  -m models/my-model.gguf \
  --system-prompt "You are a concise systems engineering assistant." \
  --ctx-size 4096

--conversation is no longer required. To end generation when the model prints a specific sequence, use a reverse prompt.

Main llama-cli flags that matter

Rather than memorizing 200 flags, focus on the ones that dominate correctness, latency, and memory.

Model and download

Context and throughput

GPU offload and hardware selection

Sampling and output quality

Example workloads with llama-cli

Summarize a file, not just a prompt

llama-cli \
  -m models/my-model.gguf \
  --system-prompt "You summarize technical documents. Output five bullets max." \
  --file ./docs/incident-report.txt \
  -n 300 \
  -st

Make results more reproducible

When you are debugging prompts, fix the seed and reduce randomness:

llama-cli \
  -m models/my-model.gguf \
  -p "Extract key risks from this design note." \
  -n 200 \
  --seed 42 \
  --temp 0.2 \
  -st

llama-server quickstart with OpenAI compatible API

llama-server is a built-in HTTP server that can expose:

• OpenAI-compatible endpoints for chat, completions, embeddings, and responses,
• a Web UI for interactive testing,
• optional monitoring endpoints for production visibility.

Start a server with a local model

llama-server \
  -m models/my-model.gguf \
  -c 4096

By default, it listens on 127.0.0.1:8080.

To bind externally (for example inside Docker or a LAN), specify host and port:

llama-server \
  -m models/my-model.gguf \
  -c 4096 \
  --host 0.0.0.0 \
  --port 8080

Optional but important server flags

If you run in containers, many settings can also be controlled through LLAMA_ARG_* environment variables.

Example API calls

Chat completions with curl

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "Give me a quick llama.cpp checklist." }
    ],
    "temperature": 0.7
  }'

Tip for real deployments: if you set --api-key, you can send it via an x-api-key header (or keep using Authorization headers depending on your gateway).

OpenAI Python client targeting llama-server

With an OpenAI-compatible server, many clients can work by changing only base_url — this is also how research agents such as lukeswade/deep-research talk to llama-server directly, without a separate adapter layer. Self-Hosted Deep Research Systems: 12 Tools Compared covers that project and others pointed at OpenAI-compatible local endpoints.

import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-no-key-required",
)

resp = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain threads vs batch size in llama.cpp."},
    ],
)

print(resp.choices[0].message.content)

Embeddings

OpenAI-compatible embeddings are exposed at /v1/embeddings, but the model must support an embedding pooling mode that is not none.

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "input": ["hello", "world"],
    "model": "GPT-4",
    "encoding_format": "float"
  }'

If you run a dedicated embedding model, consider launching the server in embeddings-only mode:

llama-server \
  -m models/Qwen3-Embedding-0.6B-Q8_0.gguf \
  --embeddings \
  --host 127.0.0.1 \
  --pooling last \
  --port 8080

or if you want to run llama-cpp with embedding model on CPU:

CUDA_VISIBLE_DEVICES="" llama-server \
  -m models/Qwen3-Embedding-0.6B-Q8_0.gguf \
  --embeddings \
  --host 127.0.0.1 \
  --pooling last \
  --port 8080

try it like this:

CUDA_VISIBLE_DEVICES="" llama-embedding \
  -m /path/to/Qwen3-Embedding-0.6B-Q8_0.gguf \
  -p "your text here" \
  --pooling last \
  --verbose-prompt

Serving multiple models from one process

The examples above bind llama-server to a single model at startup. If you need to switch between models on a per-request basis — without restarting the process — that is what router mode is for. See llama-server router mode: dynamic model switching without restarts. For a scriptable unload-all flow that frees VRAM without restarting the router, see Unload All llama.cpp Router Models Without Restarting.

Performance, monitoring, and production hardening

The FAQ question “Which llama.cpp command line options matter most for speed and memory” becomes much easier when you treat inference like a system:

• Memory ceiling is usually the first constraint (RAM on CPU, VRAM on GPU).
• Context size is a major memory multiplier.
• GPU layer offload is often the fastest path to higher tokens per second.
• Batch sizes and threads can improve throughput but can also increase memory pressure.

For a deeper, engineering-first view, see: LLM Performance in 2026: Benchmarks, Bottlenecks & Optimization.

If you want measured llama-cli-style outcomes on a 16 GB class GPU—tokens per second, VRAM, and GPU load while sweeping context (19K / 32K / 64K) across dense and MoE GGUFs—see 16 GB VRAM LLM benchmarks with llama.cpp (speed and context).

For Qwen 3.6, llama.cpp supports built-in Multi-Token Prediction speculative decoding. Turn it on with --spec-type draft-mtp; --spec-draft-n-max sets how many tokens the draft head proposes (default 3). v0.6.0 extends that same MTP path to Qwen4Exp. For the method comparison, see Speculative Decoding. For measured Qwen 3.6 numbers on 16 GB, see Qwen 3.6 MTP vs Standard on 16GB GPU.

Monitoring llama-server with Prometheus and Grafana

llama-server exposes Prometheus-compatible metrics at /metrics only when --metrics is enabled. Point a Prometheus scrape job at that path. Dashboards and alert thresholds for llama.cpp, vLLM, and TGI are in Monitor LLM Inference in Production (2026): Prometheus & Grafana for vLLM, TGI, llama.cpp.

Basic hardening checklist

When your llama-server is reachable beyond localhost:

• use --api-key (or --api-key-file) so requests are authenticated,
• avoid binding to 0.0.0.0 unless you need it,
• consider TLS via the server’s SSL flags or terminate TLS at a reverse proxy,
• restrict concurrency with --parallel to protect latency under load,
• leave --tools and --agent off. Both are experimental and expose file and shell operations over the API; enabling either also restricts CORS to localhost.

Troubleshooting quick wins

The model loads but answers are weird in chat

Chat endpoints are best when the model has a supported chat template. If outputs look unstructured, try:

• passing an explicit --system-prompt (conversation mode is already the default in llama-cli),
• verifying your model is an instruction or chat-tuned variant,
• testing using the server Web UI before wiring it into an app. v0.6.0 rebuilt that UI with a Hugging Face Hub browser, so a model can be downloaded from the page rather than only from the CLI.

Thinking models wrap the answer in reasoning traces. --reasoning off suppresses that, and --reasoning-format deepseek moves the trace into reasoning_content instead of content.

You hit out of memory

Reduce the context or choose a smaller quant:

• lower --ctx-size,
• reduce --n-gpu-layers if VRAM is the issue, or set --cpu-moe so a MoE model’s expert weights stay in RAM,
• switch to a smaller model or a more compressed quant,
• quantize the KV cache with --cache-type-k q8_0 --cache-type-v q8_0.

The context or GPU split is not what you asked for

--fit is on by default and rewrites memory settings you left unset, aiming to leave 1024 MiB free on each device (--fit-target). It will shrink the context, but never below --fit-ctx (4096), and only when you did not pass -c. It will not touch layer placement once you pass any -ngl. --fit off disables the mechanism. If logs say n_gpu_layers already set by user, your explicit -ngl is what loaded, and fit will not rescue an out-of-memory from it.

It is slow on CPU

Start with:

• --threads equal to your physical cores,
• moderate batch sizes,
• validating you installed a build that matches your machine (CPU features and backend).

References

• Official llama.cpp repository and docs
• Official CLI flags documentation
• Official llama-server documentation and API details