Meta Llama is the best-known open-weight LLM family: you download the weights and run them yourself. From the lightweight and general-purpose Llama 3.x models to the MoE-based multimodal Llama 4, this guide covers model choice, hardware sizing, running locally with Ollama, building agents and RAG, serving with vLLM, and QLoRA fine-tuning β in the order you would do them.
Llama Practical Examples
Examples of an Ollama-based chat API, RAG, code review, and a streaming web app.
View practical examples βLlama is Metaβs family of open-weight LLMs. You can download the weights and run them on your own PC, on-prem servers, or cloud GPUs, so data never has to leave for an external API, and you can fine-tune the model for your domain.
| Generation | Released | What changed |
|---|---|---|
| Llama 3.1 | July 2024 | 8B, 70B, 405B; 128K context; tool calling (function calling) |
| Llama 3.2 | September 2024 | Lightweight 1B and 3B models; 11B and 90B image-input (Vision) models |
| Llama 3.3 | December 2024 | A single 70B model with text quality close to 3.1 405B |
| Llama 4 | April 2025 | Mixture-of-experts (MoE), natively multimodal text + image, up to 10M context on Scout |
This guide covers the Llama 3.1-3.3 and Llama 4 Scout/Maverick models released as of October 2026. Llama 4 Behemoth was still training when it was announced and its weights were not released. When a new version ships, check the model card on llama.com as well.
Want to follow along from installation to data preparation, QLoRA training, evaluation, Ollama import, and API serving? See the step-by-step Llama tutorial: install, fine-tune, and serve.
Llama 4 uses MoE, so its total parameters differ from the active parameters actually computed for each token. The key to choosing: speed follows active parameters, while the memory you need follows total parameters.
| Model | Parameters | Context | Input | Ollama tag | Best for |
|---|---|---|---|---|---|
| Llama 3.2 1B / 3B | 1B / 3B | 128K | Text | llama3.2:1b, llama3.2 | Classification, summarization, edge devices, simple tool calls |
| Llama 3.1 8B | 8B | 128K | Text | llama3.1 | The default choice for local agents and RAG |
| Llama 3.2 Vision | 11B / 90B | 128K | Text + image | llama3.2-vision | Understanding documents and screenshots |
| Llama 3.3 70B | 70B | 128K | Text | llama3.3 | High-quality generation and harder reasoning on a server |
| Llama 3.1 405B | 405B | 128K | Text | llama3.1:405b | Synthetic data generation, a teacher model for distillation |
| Llama 4 Scout | 109B (17B active, 16 experts) | 10M | Text + image | llama4:scout | Very long document analysis, multimodal agents |
| Llama 4 Maverick | 400B (17B active, 128 experts) | 1M | Text + image | llama4:maverick | High-quality multimodal services on GPU servers |
If you are starting out, build the feature with Llama 3.1 8B and move only the tasks where quality falls short to 3.3 70B or Llama 4. They share the same OpenAI-compatible API, so you only change the model name.
Memory needed = weights + KV cache (context) + runtime headroom. You can estimate the weights like this:
weight memory β total parameters Γ bytes per parameter
FP16/BF16 : 2 bytes β 8B β 16 GB, 70B β 140 GB
INT8/Q8_0 : 1 byte β 8B β 8 GB, 70B β 70 GB
Q4_K_M : about 0.6 byte β 8B β 5 GB, 70B β 43 GB
MoE models are sized by total parameters, not active parameters
Llama 4 Scout (109B) Q4 β 67 GB (download size of Ollama llama4:scout)| Hardware | Realistic choice |
|---|---|
| CPU laptop (16 GB RAM) | Llama 3.2 3B, Llama 3.1 8B Q4 (slow but works) |
| 8-12 GB GPU (RTX 3060/4070 class) | Llama 3.1 8B Q4, Llama 3.2 Vision 11B Q4 |
| 24 GB GPU (RTX 3090/4090 class) | 8B at Q8/FP16, 8B with long context |
| 48 GB+ GPU / Apple Silicon 64 GB+ | Llama 3.3 70B Q4 |
| One H100 80 GB | Llama 4 Scout INT4 (per Metaβs guidance) |
| Several H100s (one host) | Llama 4 Maverick, 70B FP16 |
The KV cache grows with context length. Even if a model supports a 10M context, the length you can actually use is set by the memory you have left. For long documents, decide the context length you need first and make sure you have the memory for it.
Llama is not released under an OSI open-source license such as Apache 2.0, but under Metaβs Llama Community License. Most companies can use it commercially for free, but check these terms before shipping it in a product:
The meta-llama/* repositories on Hugging Face are gated: accept the license on the model page, then register a token with hf auth login.
Ollama is the easiest way to run an LLM locally. Pull a model and run it right away; it serves a REST and OpenAI-compatible API on port 11434.
# Install (macOS/Linux) β on Windows, download the installer from ollama.com
curl -fsSL https://ollama.com/install.sh | sh
# Download and run models
ollama pull llama3.1 # 8B (about 4.9 GB, Q4_K_M)
ollama pull llama3.2 # 3B, lightweight
ollama pull llama3.3 # 70B (about 43 GB)
ollama pull llama4:scout # Llama 4 Scout (about 67 GB)
ollama run llama3.1 # interactive REPL
# Run the server (the desktop app starts it automatically)
ollama serve # default port 11434
# Manage models
ollama list # installed models
ollama ps # loaded models and their GPU/CPU split
ollama show llama3.1 # context length, quantization, capabilities
ollama rm llama3.1 # delete a modelBundle the settings you use often into a Modelfile and save it under your own model name.
FROM llama3.1
# Inference parameters
PARAMETER temperature 0.2
PARAMETER num_ctx 16384
# Fixed system prompt
SYSTEM """You are a support assistant that answers from the team's DevOps runbooks.
If the runbooks do not cover it, say you don't know."""ollama create devops-assistant -f Modelfile
ollama run devops-assistantOllama uses Metal on macOS and NVIDIA CUDA or AMD ROCm on Linux and Windows automatically. If the PROCESSOR column of ollama ps shows a CPU share, the model did not fit entirely in VRAM.
Ollama runs with a default sized to your GPU memory, not the modelβs maximum context. If RAG or a long conversation suddenly forgets earlier content, this is the first setting to suspect.
| VRAM | Ollama default context |
|---|---|
| Under 24 GiB | 4K |
| 24-48 GiB | 32K |
| 48 GiB or more | 256K |
# Change the server-wide default
OLLAMA_CONTEXT_LENGTH=32768 ollama servefrom pathlib import Path
from ollama import chat
long_document = Path("runbook.md").read_text(encoding="utf-8")
# Set it per request (Ollama native API)
resp = chat(
model="llama3.1",
messages=[{"role": "user", "content": long_document + "\n\nSummarize the document above"}],
options={"num_ctx": 32768},
)
print(resp.message.content)The OpenAI-compatible endpoint (/v1) cannot take num_ctx per request. Set it with the server environment variable or PARAMETER num_ctx in a Modelfile.
Ollama and vLLM are compatible with the OpenAI Chat Completions API. Change only base_url and the model name in existing OpenAI SDK code to switch to Llama.
import asyncio
from openai import OpenAI, AsyncOpenAI
BASE_URL = "http://localhost:11434/v1" # for vLLM: http://localhost:8000/v1
MODEL = "llama3.1"
# ββ Synchronous call βββββββββββββββββββββββββββββ
client = OpenAI(base_url=BASE_URL, api_key="ollama") # Ollama does not check the key
resp = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": "You are a senior Python developer."},
{"role": "user", "content": "Explain the difference between asyncio.gather and asyncio.wait"},
],
temperature=0.7,
max_tokens=1024,
)
print(resp.choices[0].message.content)
print(resp.usage) # prompt_tokens / completion_tokens β for cost and latency monitoring
# ββ Streaming ββββββββββββββββββββββββββββββββββββ
async def stream_response(prompt: str) -> None:
aclient = AsyncOpenAI(base_url=BASE_URL, api_key="ollama")
stream = await aclient.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
stream=True,
)
async for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
asyncio.run(stream_response("Explain common Python async patterns"))Llama 3.1 and later (including 3.2 1B and 3B) support tool calling. Pass tool schemas and the model generates which tool to call and its arguments as JSON; your code runs the tool and sends the result back, in a loop. Smaller models choose tools less reliably, so 8B or larger is recommended.
import json
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
MODEL = "llama3.1"
# ββ Tool functions βββββββββββββββββββββββββββββββ
def get_weather(city: str, unit: str = "celsius") -> dict:
"""Look up the weather for a city (call a weather API in a real implementation)"""
return {"city": city, "temp": 23, "unit": unit, "condition": "rain", "rain_probability": 0.8}
def search_docs(query: str, top_k: int = 3) -> dict:
"""Search internal documents (query a vector DB in a real implementation)"""
return {"results": [f"Document {i + 1} about {query}" for i in range(top_k)]}
TOOLS = {"get_weather": get_weather, "search_docs": search_docs}
TOOL_SCHEMAS = [
{"type": "function", "function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {"type": "object",
"properties": {
"city": {"type": "string", "description": "City name (e.g. Seattle)"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
}, "required": ["city"]},
}},
{"type": "function", "function": {
"name": "search_docs",
"description": "Search the internal knowledge base for relevant documents",
"parameters": {"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query"},
"top_k": {"type": "integer", "description": "Number of results to return"},
}, "required": ["query"]},
}},
]
def call_tool(name: str, raw_args: str) -> str:
"""Validate and run a call the model produced β return bad calls as errors so the model can fix them itself"""
if name not in TOOLS:
return json.dumps({"error": f"unknown tool: {name}"})
try:
args = json.loads(raw_args or "{}")
return json.dumps(TOOLS[name](**args))
except (json.JSONDecodeError, TypeError) as e:
return json.dumps({"error": f"bad arguments: {e}"})
# ββ Agent loop βββββββββββββββββββββββββββββββββββ
def run_agent(user_query: str, max_steps: int = 5) -> str:
messages = [
{"role": "system", "content": "You are a helpful AI assistant. Use tools only when you need them."},
{"role": "user", "content": user_query},
]
for _ in range(max_steps):
resp = client.chat.completions.create(
model=MODEL, messages=messages, tools=TOOL_SCHEMAS, tool_choice="auto", temperature=0,
)
msg = resp.choices[0].message
messages.append(msg.model_dump(exclude_none=True)) # store the response object as a dict
if not msg.tool_calls: # no tool call means a final answer
return msg.content
for tc in msg.tool_calls:
result = call_tool(tc.function.name, tc.function.arguments)
print(f"[tool] {tc.function.name}({tc.function.arguments}) -> {result}")
messages.append({"role": "tool", "tool_call_id": tc.id, "content": result})
return "Reached the step limit. Try splitting the question into smaller parts."
print(run_agent("What's the weather in Seattle, and do I need an umbrella?"))In production, always set a loop limit, per-tool timeouts, and call logs (tool name, arguments, result, latency). Most agent quality problems are diagnosed from those logs.
Pass a JSON schema to Ollamaβs format and decoding is constrained to output that matches it. That is far more reliable than asking for JSON in the prompt, and Pydantic validates the types too.
uv add ollama pydanticfrom typing import Literal
from ollama import chat
from pydantic import BaseModel, Field
class SentimentResult(BaseModel):
sentiment: Literal["positive", "negative", "neutral"]
score: float = Field(ge=0, le=1, description="confidence of the judgment")
reason: str = Field(description="one-sentence rationale")
keywords: list[str]
def analyze_sentiment(text: str) -> SentimentResult:
resp = chat(
model="llama3.1",
messages=[
{"role": "system", "content": "You analyze the sentiment of customer reviews."},
{"role": "user", "content": text},
],
format=SentimentResult.model_json_schema(), # enforce the output shape with the schema
options={"temperature": 0},
)
return SentimentResult.model_validate_json(resp.message.content)
result = analyze_sentiment("Shipping was late and the quality was below what I expected. Disappointed.")
print(result.sentiment, f"{result.score:.0%}", result.reason, result.keywords)On the OpenAI-compatible endpoint, use response_format for the same feature. vLLM also supports JSON-schema-based structured output (guided decoding).
Llama 3.2 Vision (11B, 90B) and Llama 4 take images as input and answer in text. You can handle receipts, screenshots, diagrams, and error screens locally.
from ollama import chat
resp = chat(
model="llama3.2-vision", # with GPU headroom: "llama4:scout"
messages=[{
"role": "user",
"content": "What on this error screen looks like the cause, and what should I check?",
"images": ["./error-screenshot.png"], # a file path, bytes, or base64 all work
}],
)
print(resp.message.content)Use Llama as the generator and Chroma as the vector store to build RAG that runs entirely locally, with no external API. Internal documents never leave your machine.
uv add langchain-ollama langchain-chroma langchain-huggingface langchain-text-splitters sentence-transformersimport torch
from langchain_chroma import Chroma
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_ollama import ChatOllama
from langchain_text_splitters import RecursiveCharacterTextSplitter
# ββ Embedding model (local, multilingual) ββββββββ
embeddings = HuggingFaceEmbeddings(
model_name="BAAI/bge-m3",
model_kwargs={"device": "cuda" if torch.cuda.is_available() else "cpu"},
encode_kwargs={"normalize_embeddings": True},
)
# ββ Split and index documents ββββββββββββββββββββ
raw_docs = [
"AIDevOps is a learning platform for AI, Cloud, and DevOps.",
"k6 is a modern load testing tool where scenarios are written in JavaScript.",
"JMeter is a Java-based load testing tool with a GUI and broad protocol support.",
]
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
docs = splitter.create_documents(raw_docs)
vectordb = Chroma.from_documents(docs, embeddings, persist_directory="./chroma_db")
retriever = vectordb.as_retriever(search_kwargs={"k": 3})
def format_docs(found) -> str:
return "\n\n".join(d.page_content for d in found)
# ββ RAG chain (LCEL) βββββββββββββββββββββββββββββ
llm = ChatOllama(model="llama3.1", temperature=0.1, num_ctx=8192)
prompt = ChatPromptTemplate.from_template("""Answer the question using only the context below.
If the context does not contain the answer, say you don't know.
Context:
{context}
Question: {question}""")
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
print(rag_chain.invoke("How do k6 and JMeter differ?"))Join only the page text, as format_docs does, before putting search results in the prompt. Passing Document objects directly also inserts their metadata strings and wastes tokens. For evaluating retrieval quality, see the Agent Evaluation guide.
The same Llama weights behave very differently depending on the engine: setup effort, concurrent throughput, and supported hardware all change.
| Engine | Strengths | Weaknesses | Good fit |
|---|---|---|---|
| Ollama | Easiest install and model management; CPU and Apple Silicon support | Limited concurrent throughput | Personal development, small internal tools |
| llama.cpp | GGUF quantization, CPU and edge devices, fine-grained tuning | You build and configure it yourself | Embedded, low-spec servers |
| vLLM | High throughput with PagedAttention and continuous batching, multi-GPU | Needs an NVIDIA/AMD GPU server | Production API servers |
| Hugging Face TGI | Integrated with the Hugging Face ecosystem | Fewer options than vLLM | Deployments on HF infrastructure |
vLLM manages the KV cache in pages (PagedAttention) and batches continuously, which gives high throughput under many concurrent requests. It exposes an OpenAI-compatible API, so the code you wrote against Ollama in development connects unchanged.
uv pip install vllm
# Llama 3.1 8B + tool calling (uses the templates in the vLLM repo's examples/)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--port 8000 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--tool-call-parser llama3_json \
--chat-template examples/tool_chat_template_llama3.1_json.jinja
# Llama 4 Scout β BF16 weights alone are about 218 GB, so split them across GPUs with tensor parallelism
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
--tensor-parallel-size 8 \
--max-model-len 131072 \
--enable-auto-tool-choice \
--tool-call-parser llama4_pythonic \
--chat-template examples/tool_chat_template_llama4_pythonic.jinjaservices:
vllm:
image: vllm/vllm-openai:latest
command: >
--model meta-llama/Llama-3.1-8B-Instruct
--max-model-len 32768
--gpu-memory-utilization 0.90
environment:
HF_TOKEN: ${HF_TOKEN} # for downloading gated models
ports:
- "8000:8000"
volumes:
- hf-cache:/root/.cache/huggingface # avoid re-downloading the model on restart
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
hf-cache:import asyncio
import httpx
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="none")
MODEL = "meta-llama/Llama-3.1-8B-Instruct"
# ββ Concurrent requests β vLLM batches them on the server β
async def batch_inference(prompts: list[str], concurrency: int = 16) -> list[str]:
sem = asyncio.Semaphore(concurrency)
async def one(p: str) -> str:
async with sem:
r = await client.chat.completions.create(
model=MODEL, messages=[{"role": "user", "content": p}], max_tokens=512,
)
return r.choices[0].message.content
return await asyncio.gather(*(one(p) for p in prompts))
# ββ Check Prometheus metrics βββββββββββββββββββββ
async def check_metrics() -> None:
async with httpx.AsyncClient() as http:
text = (await http.get("http://localhost:8000/metrics")).text
for line in text.splitlines():
if line.startswith(("vllm:num_requests_running", "vllm:num_requests_waiting", "vllm:gpu_cache_usage_perc")):
print(line)
print(asyncio.run(batch_inference(["What is Python?", "What is Go?", "What is Rust?"])))
asyncio.run(check_metrics())Scrape /metrics with Prometheus and dashboard waiting requests, KV cache usage, and time to first token (TTFT). Before launch, ramp up concurrent users with k6 and measure latency and error rate to decide how many GPUs you need.
Fix tone, output format, and domain vocabulary that prompting and RAG cannot fix with fine-tuning. QLoRA loads the model in 4-bit and trains only small LoRA adapters, so Llama 3.1 8B trains on a single GPU with around 16 GB. 70B needs a GPU with 48 GB or more.
uv add "trl>=1.0" peft datasets bitsandbytes accelerate{"messages": [{"role": "system", "content": "Incident-response assistant"}, {"role": "user", "content": "API 5xx rate just spiked"}, {"role": "assistant", "content": "1) Check for a recent deploy 2) Check upstream latency 3) ..."}]}
{"messages": [{"role": "user", "content": "What does DB connection pool exhaustion look like?"}, {"role": "assistant", "content": "Request wait times climb and ... "}]}import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
# Conversational data in messages format β the Llama chat template is applied automatically
dataset = load_dataset("json", data_files="train.jsonl", split="train")
trainer = SFTTrainer(
model=MODEL_ID,
train_dataset=dataset,
# ββ QLoRA: load in 4-bit ββββββββββββββββββββ
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
),
# ββ LoRA adapters βββββββββββββββββββββββββββ
peft_config=LoraConfig(
r=16, # rank: higher = more capacity, more memory
lora_alpha=32,
lora_dropout=0.05,
target_modules="all-linear", # every linear layer in attention + MLP
task_type="CAUSAL_LM",
),
args=SFTConfig(
output_dir="./llama-qlora",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=1e-4, # adapters train with a higher learning rate than full fine-tuning
max_length=2048,
logging_steps=10,
save_steps=100,
model_init_kwargs={"dtype": torch.bfloat16},
),
)
trainer.train()
trainer.save_model() # saves only the LoRA adapter (tens of MB)import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
# Merge the adapter into the base model and save one model β for vLLM serving or GGUF conversion
model = AutoPeftModelForCausalLM.from_pretrained("./llama-qlora", dtype=torch.bfloat16)
merged = model.merge_and_unload()
merged.save_pretrained("./llama-merged")
AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct").save_pretrained("./llama-merged")Run the same evaluation set before and after fine-tuning. It is common for a specific task to improve while general ability or safety drops. If you distribute the fine-tuned model, the license requires its name to start with "Llama".
The full flow β cleaning data, reading training logs, comparing before and after, GGUF conversion, and Ollama import β continues in steps 6-9 of the hands-on tutorial.
| Symptom | Cause | Fix |
|---|---|---|
| Responses are very slow | The model does not fit in VRAM, so part of it runs on the CPU | Check the CPU share with ollama ps β pick a smaller model or quantization, shorten the context |
| Forgets the start of long documents | Input beyond the default context (4K, etc.) is cut off | Raise OLLAMA_CONTEXT_LENGTH or num_ctx |
| Answers in prose instead of calling tools | Small model, vague tool descriptions, high temperature | Use 8B or larger, make tool descriptions specific, temperature=0 |
| JSON fails to parse | The format is only requested in the prompt | Pass a JSON schema to format/response_format |
| CUDA OOM when vLLM starts | --max-model-len is too large to reserve the KV cache | Lower --max-model-len, adjust --gpu-memory-utilization, increase tensor parallelism |
| Hugging Face 401/403 | License not accepted for the gated model, or no token | Accept it on the model page, then hf auth login |
Most services can use Llama commercially for free under the Llama Community License. Services with more than 700 million monthly active users on the modelβs release date must request a separate license from Meta, and there are conditions such as displaying "Built with Llama" and following the Acceptable Use Policy, so read the license for the version you use.
On a local PC or a single consumer GPU, Llama 3.1 8B or Llama 3.2 3B is realistic. Llama 4 Scout is fast because only 17B parameters are active per token, but its full weights are 109B, so it needs that much memory. Choose Llama 4 if you need image input or very long context and have a GPU server; for a 70B-class text server, look at Llama 3.3 70B first.
With 4-bit (Q4_K_M) quantization the weights are about 5 GB; add the KV cache for the context and it runs on an 8 GB GPU. At 16-bit the weights alone are about 16 GB. It also runs on CPU and system memory without a GPU, but responses are much slower.
Ollama is easy to install and manage models with, which suits personal development and small internal services. vLLM uses PagedAttention and continuous batching for high throughput under many concurrent requests, which suits production API servers. Both expose an OpenAI-compatible API, so a common setup is Ollama for development and vLLM for production with the same code.
Ollama runs with a default context length sized to your GPU memory, not the modelβs maximum. Below 24 GiB of VRAM the default is 4K tokens, so longer input gets cut off at the start. Raise it with the OLLAMA_CONTEXT_LENGTH environment variable or the num_ctx option on the request.