Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
Open-Weight LLM Guide for AI Agent Development

πŸ¦™ Llama Complete Guide

Visitors

Meta Llama is the best-known open-weight LLM family: you download the weights and run them yourself. From the lightweight and general-purpose Llama 3.x models to the MoE-based multimodal Llama 4, this guide covers model choice, hardware sizing, running locally with Ollama, building agents and RAG, serving with vLLM, and QLoRA fine-tuning β€” in the order you would do them.

← Back to AIDevOps home
πŸ¦™

Llama Practical Examples

Examples of an Ollama-based chat API, RAG, code review, and a streaming web app.

View practical examples β†’

Contents

0 / 16
  1. Llama at a glance
  2. Model lineup & how to choose
  3. VRAM sizing & quantization
  4. License checklist
  5. Running locally with Ollama
  6. Context length
  7. OpenAI-compatible API
  8. Function Calling Agent
  9. Structured JSON output
  10. Image input (Vision)
  11. RAG pipeline
  12. Inference engines compared
  13. vLLM production server
  14. QLoRA fine-tuning
  15. Troubleshooting
  16. FAQ
Contents 16 sections
  1. Llama at a glance
  2. Model lineup & how to choose
  3. VRAM sizing & quantization
  4. License checklist
  5. Running locally with Ollama
  6. Context length
  7. OpenAI-compatible API
  8. Function Calling Agent
  9. Structured JSON output
  10. Image input (Vision)
  11. RAG pipeline
  12. Inference engines compared
  13. vLLM production server
  14. QLoRA fine-tuning
  15. Troubleshooting
  16. FAQ

Llama at a glance

Llama is Meta’s family of open-weight LLMs. You can download the weights and run them on your own PC, on-prem servers, or cloud GPUs, so data never has to leave for an external API, and you can fine-tune the model for your domain.

GenerationReleasedWhat changed
Llama 3.1July 20248B, 70B, 405B; 128K context; tool calling (function calling)
Llama 3.2September 2024Lightweight 1B and 3B models; 11B and 90B image-input (Vision) models
Llama 3.3December 2024A single 70B model with text quality close to 3.1 405B
Llama 4April 2025Mixture-of-experts (MoE), natively multimodal text + image, up to 10M context on Scout
ℹ️

This guide covers the Llama 3.1-3.3 and Llama 4 Scout/Maverick models released as of October 2026. Llama 4 Behemoth was still training when it was announced and its weights were not released. When a new version ships, check the model card on llama.com as well.

πŸ’‘

Want to follow along from installation to data preparation, QLoRA training, evaluation, Ollama import, and API serving? See the step-by-step Llama tutorial: install, fine-tune, and serve.

Model lineup & how to choose

Llama 4 uses MoE, so its total parameters differ from the active parameters actually computed for each token. The key to choosing: speed follows active parameters, while the memory you need follows total parameters.

ModelParametersContextInputOllama tagBest for
Llama 3.2 1B / 3B1B / 3B128KTextllama3.2:1b, llama3.2Classification, summarization, edge devices, simple tool calls
Llama 3.1 8B8B128KTextllama3.1The default choice for local agents and RAG
Llama 3.2 Vision11B / 90B128KText + imagellama3.2-visionUnderstanding documents and screenshots
Llama 3.3 70B70B128KTextllama3.3High-quality generation and harder reasoning on a server
Llama 3.1 405B405B128KTextllama3.1:405bSynthetic data generation, a teacher model for distillation
Llama 4 Scout109B (17B active, 16 experts)10MText + imagellama4:scoutVery long document analysis, multimodal agents
Llama 4 Maverick400B (17B active, 128 experts)1MText + imagellama4:maverickHigh-quality multimodal services on GPU servers
πŸ’‘

If you are starting out, build the feature with Llama 3.1 8B and move only the tasks where quality falls short to 3.3 70B or Llama 4. They share the same OpenAI-compatible API, so you only change the model name.

VRAM sizing & quantization

Memory needed = weights + KV cache (context) + runtime headroom. You can estimate the weights like this:

TEXT
weight memory β‰ˆ total parameters Γ— bytes per parameter

  FP16/BF16 : 2 bytes        β†’  8B β‰ˆ 16 GB,  70B β‰ˆ 140 GB
  INT8/Q8_0 : 1 byte         β†’  8B β‰ˆ  8 GB,  70B β‰ˆ  70 GB
  Q4_K_M    : about 0.6 byte β†’  8B β‰ˆ  5 GB,  70B β‰ˆ  43 GB

MoE models are sized by total parameters, not active parameters
  Llama 4 Scout (109B) Q4 β‰ˆ 67 GB  (download size of Ollama llama4:scout)
HardwareRealistic choice
CPU laptop (16 GB RAM)Llama 3.2 3B, Llama 3.1 8B Q4 (slow but works)
8-12 GB GPU (RTX 3060/4070 class)Llama 3.1 8B Q4, Llama 3.2 Vision 11B Q4
24 GB GPU (RTX 3090/4090 class)8B at Q8/FP16, 8B with long context
48 GB+ GPU / Apple Silicon 64 GB+Llama 3.3 70B Q4
One H100 80 GBLlama 4 Scout INT4 (per Meta’s guidance)
Several H100s (one host)Llama 4 Maverick, 70B FP16
⚠️

The KV cache grows with context length. Even if a model supports a 10M context, the length you can actually use is set by the memory you have left. For long documents, decide the context length you need first and make sure you have the memory for it.

License checklist

Llama is not released under an OSI open-source license such as Apache 2.0, but under Meta’s Llama Community License. Most companies can use it commercially for free, but check these terms before shipping it in a product:

  • 700 million monthly active users β€” if your products had more than 700 million MAU on the model’s release date, you must request a separate license from Meta.
  • Attribution β€” prominently display "Built with Llama" on a related website, UI, or documentation.
  • Derivative model names β€” a model you fine-tune from Llama and distribute must start its name with "Llama".
  • Acceptable Use Policy β€” follow the list of prohibited uses; multimodal models may carry region-specific terms, so read the policy for the version you use.
ℹ️

The meta-llama/* repositories on Hugging Face are gated: accept the license on the model page, then register a token with hf auth login.

Running locally with Ollama

Ollama is the easiest way to run an LLM locally. Pull a model and run it right away; it serves a REST and OpenAI-compatible API on port 11434.

BASH
# Install (macOS/Linux) β€” on Windows, download the installer from ollama.com
curl -fsSL https://ollama.com/install.sh | sh

# Download and run models
ollama pull llama3.1          # 8B (about 4.9 GB, Q4_K_M)
ollama pull llama3.2          # 3B, lightweight
ollama pull llama3.3          # 70B (about 43 GB)
ollama pull llama4:scout      # Llama 4 Scout (about 67 GB)
ollama run llama3.1           # interactive REPL

# Run the server (the desktop app starts it automatically)
ollama serve                  # default port 11434

# Manage models
ollama list                   # installed models
ollama ps                     # loaded models and their GPU/CPU split
ollama show llama3.1          # context length, quantization, capabilities
ollama rm llama3.1            # delete a model

Bundle the settings you use often into a Modelfile and save it under your own model name.

ModelfileTEXT
FROM llama3.1

# Inference parameters
PARAMETER temperature 0.2
PARAMETER num_ctx 16384

# Fixed system prompt
SYSTEM """You are a support assistant that answers from the team's DevOps runbooks.
If the runbooks do not cover it, say you don't know."""
BASH
ollama create devops-assistant -f Modelfile
ollama run devops-assistant
ℹ️

Ollama uses Metal on macOS and NVIDIA CUDA or AMD ROCm on Linux and Windows automatically. If the PROCESSOR column of ollama ps shows a CPU share, the model did not fit entirely in VRAM.

Context length

Ollama runs with a default sized to your GPU memory, not the model’s maximum context. If RAG or a long conversation suddenly forgets earlier content, this is the first setting to suspect.

VRAMOllama default context
Under 24 GiB4K
24-48 GiB32K
48 GiB or more256K
BASH
# Change the server-wide default
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
num_ctx.pyPYTHON
from pathlib import Path
from ollama import chat

long_document = Path("runbook.md").read_text(encoding="utf-8")

# Set it per request (Ollama native API)
resp = chat(
    model="llama3.1",
    messages=[{"role": "user", "content": long_document + "\n\nSummarize the document above"}],
    options={"num_ctx": 32768},
)
print(resp.message.content)
πŸ’‘

The OpenAI-compatible endpoint (/v1) cannot take num_ctx per request. Set it with the server environment variable or PARAMETER num_ctx in a Modelfile.

OpenAI-compatible API

Ollama and vLLM are compatible with the OpenAI Chat Completions API. Change only base_url and the model name in existing OpenAI SDK code to switch to Llama.

llama_client.pyPYTHON
import asyncio
from openai import OpenAI, AsyncOpenAI

BASE_URL = "http://localhost:11434/v1"   # for vLLM: http://localhost:8000/v1
MODEL = "llama3.1"

# ── Synchronous call ─────────────────────────────
client = OpenAI(base_url=BASE_URL, api_key="ollama")   # Ollama does not check the key

resp = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a senior Python developer."},
        {"role": "user",   "content": "Explain the difference between asyncio.gather and asyncio.wait"},
    ],
    temperature=0.7,
    max_tokens=1024,
)
print(resp.choices[0].message.content)
print(resp.usage)   # prompt_tokens / completion_tokens β€” for cost and latency monitoring

# ── Streaming ────────────────────────────────────
async def stream_response(prompt: str) -> None:
    aclient = AsyncOpenAI(base_url=BASE_URL, api_key="ollama")
    stream = await aclient.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
    print()

asyncio.run(stream_response("Explain common Python async patterns"))

Function Calling Agent

Llama 3.1 and later (including 3.2 1B and 3B) support tool calling. Pass tool schemas and the model generates which tool to call and its arguments as JSON; your code runs the tool and sends the result back, in a loop. Smaller models choose tools less reliably, so 8B or larger is recommended.

llama_agent.pyPYTHON
import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
MODEL = "llama3.1"

# ── Tool functions ───────────────────────────────
def get_weather(city: str, unit: str = "celsius") -> dict:
    """Look up the weather for a city (call a weather API in a real implementation)"""
    return {"city": city, "temp": 23, "unit": unit, "condition": "rain", "rain_probability": 0.8}

def search_docs(query: str, top_k: int = 3) -> dict:
    """Search internal documents (query a vector DB in a real implementation)"""
    return {"results": [f"Document {i + 1} about {query}" for i in range(top_k)]}

TOOLS = {"get_weather": get_weather, "search_docs": search_docs}

TOOL_SCHEMAS = [
    {"type": "function", "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {"type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name (e.g. Seattle)"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            }, "required": ["city"]},
    }},
    {"type": "function", "function": {
        "name": "search_docs",
        "description": "Search the internal knowledge base for relevant documents",
        "parameters": {"type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search query"},
                "top_k": {"type": "integer", "description": "Number of results to return"},
            }, "required": ["query"]},
    }},
]

def call_tool(name: str, raw_args: str) -> str:
    """Validate and run a call the model produced β€” return bad calls as errors so the model can fix them itself"""
    if name not in TOOLS:
        return json.dumps({"error": f"unknown tool: {name}"})
    try:
        args = json.loads(raw_args or "{}")
        return json.dumps(TOOLS[name](**args))
    except (json.JSONDecodeError, TypeError) as e:
        return json.dumps({"error": f"bad arguments: {e}"})

# ── Agent loop ───────────────────────────────────
def run_agent(user_query: str, max_steps: int = 5) -> str:
    messages = [
        {"role": "system", "content": "You are a helpful AI assistant. Use tools only when you need them."},
        {"role": "user", "content": user_query},
    ]
    for _ in range(max_steps):
        resp = client.chat.completions.create(
            model=MODEL, messages=messages, tools=TOOL_SCHEMAS, tool_choice="auto", temperature=0,
        )
        msg = resp.choices[0].message
        messages.append(msg.model_dump(exclude_none=True))   # store the response object as a dict

        if not msg.tool_calls:          # no tool call means a final answer
            return msg.content

        for tc in msg.tool_calls:
            result = call_tool(tc.function.name, tc.function.arguments)
            print(f"[tool] {tc.function.name}({tc.function.arguments}) -> {result}")
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": result})
    return "Reached the step limit. Try splitting the question into smaller parts."

print(run_agent("What's the weather in Seattle, and do I need an umbrella?"))
πŸ’‘

In production, always set a loop limit, per-tool timeouts, and call logs (tool name, arguments, result, latency). Most agent quality problems are diagnosed from those logs.

Structured JSON output

Pass a JSON schema to Ollama’s format and decoding is constrained to output that matches it. That is far more reliable than asking for JSON in the prompt, and Pydantic validates the types too.

BASH
uv add ollama pydantic
structured_llama.pyPYTHON
from typing import Literal
from ollama import chat
from pydantic import BaseModel, Field

class SentimentResult(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    score: float = Field(ge=0, le=1, description="confidence of the judgment")
    reason: str = Field(description="one-sentence rationale")
    keywords: list[str]

def analyze_sentiment(text: str) -> SentimentResult:
    resp = chat(
        model="llama3.1",
        messages=[
            {"role": "system", "content": "You analyze the sentiment of customer reviews."},
            {"role": "user", "content": text},
        ],
        format=SentimentResult.model_json_schema(),   # enforce the output shape with the schema
        options={"temperature": 0},
    )
    return SentimentResult.model_validate_json(resp.message.content)

result = analyze_sentiment("Shipping was late and the quality was below what I expected. Disappointed.")
print(result.sentiment, f"{result.score:.0%}", result.reason, result.keywords)
ℹ️

On the OpenAI-compatible endpoint, use response_format for the same feature. vLLM also supports JSON-schema-based structured output (guided decoding).

Image input (Vision)

Llama 3.2 Vision (11B, 90B) and Llama 4 take images as input and answer in text. You can handle receipts, screenshots, diagrams, and error screens locally.

vision_llama.pyPYTHON
from ollama import chat

resp = chat(
    model="llama3.2-vision",          # with GPU headroom: "llama4:scout"
    messages=[{
        "role": "user",
        "content": "What on this error screen looks like the cause, and what should I check?",
        "images": ["./error-screenshot.png"],   # a file path, bytes, or base64 all work
    }],
)
print(resp.message.content)

RAG pipeline

Use Llama as the generator and Chroma as the vector store to build RAG that runs entirely locally, with no external API. Internal documents never leave your machine.

BASH
uv add langchain-ollama langchain-chroma langchain-huggingface langchain-text-splitters sentence-transformers
local_rag.pyPYTHON
import torch
from langchain_chroma import Chroma
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_ollama import ChatOllama
from langchain_text_splitters import RecursiveCharacterTextSplitter

# ── Embedding model (local, multilingual) ────────
embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-m3",
    model_kwargs={"device": "cuda" if torch.cuda.is_available() else "cpu"},
    encode_kwargs={"normalize_embeddings": True},
)

# ── Split and index documents ────────────────────
raw_docs = [
    "AIDevOps is a learning platform for AI, Cloud, and DevOps.",
    "k6 is a modern load testing tool where scenarios are written in JavaScript.",
    "JMeter is a Java-based load testing tool with a GUI and broad protocol support.",
]
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
docs = splitter.create_documents(raw_docs)
vectordb = Chroma.from_documents(docs, embeddings, persist_directory="./chroma_db")
retriever = vectordb.as_retriever(search_kwargs={"k": 3})

def format_docs(found) -> str:
    return "\n\n".join(d.page_content for d in found)

# ── RAG chain (LCEL) ─────────────────────────────
llm = ChatOllama(model="llama3.1", temperature=0.1, num_ctx=8192)

prompt = ChatPromptTemplate.from_template("""Answer the question using only the context below.
If the context does not contain the answer, say you don't know.

Context:
{context}

Question: {question}""")

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(rag_chain.invoke("How do k6 and JMeter differ?"))
πŸ’‘

Join only the page text, as format_docs does, before putting search results in the prompt. Passing Document objects directly also inserts their metadata strings and wastes tokens. For evaluating retrieval quality, see the Agent Evaluation guide.

Inference engines compared

The same Llama weights behave very differently depending on the engine: setup effort, concurrent throughput, and supported hardware all change.

EngineStrengthsWeaknessesGood fit
OllamaEasiest install and model management; CPU and Apple Silicon supportLimited concurrent throughputPersonal development, small internal tools
llama.cppGGUF quantization, CPU and edge devices, fine-grained tuningYou build and configure it yourselfEmbedded, low-spec servers
vLLMHigh throughput with PagedAttention and continuous batching, multi-GPUNeeds an NVIDIA/AMD GPU serverProduction API servers
Hugging Face TGIIntegrated with the Hugging Face ecosystemFewer options than vLLMDeployments on HF infrastructure

vLLM production server

vLLM manages the KV cache in pages (PagedAttention) and batches continuously, which gives high throughput under many concurrent requests. It exposes an OpenAI-compatible API, so the code you wrote against Ollama in development connects unchanged.

BASH
uv pip install vllm

# Llama 3.1 8B + tool calling (uses the templates in the vLLM repo's examples/)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json \
  --chat-template examples/tool_chat_template_llama3.1_json.jinja

# Llama 4 Scout β€” BF16 weights alone are about 218 GB, so split them across GPUs with tensor parallelism
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser llama4_pythonic \
  --chat-template examples/tool_chat_template_llama4_pythonic.jinja
docker-compose.ymlYAML
services:
  vllm:
    image: vllm/vllm-openai:latest
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --max-model-len 32768
      --gpu-memory-utilization 0.90
    environment:
      HF_TOKEN: ${HF_TOKEN}            # for downloading gated models
    ports:
      - "8000:8000"
    volumes:
      - hf-cache:/root/.cache/huggingface   # avoid re-downloading the model on restart
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  hf-cache:
vllm_client.pyPYTHON
import asyncio
import httpx
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="none")
MODEL = "meta-llama/Llama-3.1-8B-Instruct"

# ── Concurrent requests β€” vLLM batches them on the server ─
async def batch_inference(prompts: list[str], concurrency: int = 16) -> list[str]:
    sem = asyncio.Semaphore(concurrency)

    async def one(p: str) -> str:
        async with sem:
            r = await client.chat.completions.create(
                model=MODEL, messages=[{"role": "user", "content": p}], max_tokens=512,
            )
            return r.choices[0].message.content

    return await asyncio.gather(*(one(p) for p in prompts))

# ── Check Prometheus metrics ─────────────────────
async def check_metrics() -> None:
    async with httpx.AsyncClient() as http:
        text = (await http.get("http://localhost:8000/metrics")).text
        for line in text.splitlines():
            if line.startswith(("vllm:num_requests_running", "vllm:num_requests_waiting", "vllm:gpu_cache_usage_perc")):
                print(line)

print(asyncio.run(batch_inference(["What is Python?", "What is Go?", "What is Rust?"])))
asyncio.run(check_metrics())
πŸ’‘

Scrape /metrics with Prometheus and dashboard waiting requests, KV cache usage, and time to first token (TTFT). Before launch, ramp up concurrent users with k6 and measure latency and error rate to decide how many GPUs you need.

QLoRA fine-tuning

Fix tone, output format, and domain vocabulary that prompting and RAG cannot fix with fine-tuning. QLoRA loads the model in 4-bit and trains only small LoRA adapters, so Llama 3.1 8B trains on a single GPU with around 16 GB. 70B needs a GPU with 48 GB or more.

BASH
uv add "trl>=1.0" peft datasets bitsandbytes accelerate
train.jsonlJSON
{"messages": [{"role": "system", "content": "Incident-response assistant"}, {"role": "user", "content": "API 5xx rate just spiked"}, {"role": "assistant", "content": "1) Check for a recent deploy 2) Check upstream latency 3) ..."}]}
{"messages": [{"role": "user", "content": "What does DB connection pool exhaustion look like?"}, {"role": "assistant", "content": "Request wait times climb and ... "}]}
finetune_llama.pyPYTHON
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

# Conversational data in messages format β€” the Llama chat template is applied automatically
dataset = load_dataset("json", data_files="train.jsonl", split="train")

trainer = SFTTrainer(
    model=MODEL_ID,
    train_dataset=dataset,
    # ── QLoRA: load in 4-bit ────────────────────
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
    ),
    # ── LoRA adapters ───────────────────────────
    peft_config=LoraConfig(
        r=16,                         # rank: higher = more capacity, more memory
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules="all-linear",  # every linear layer in attention + MLP
        task_type="CAUSAL_LM",
    ),
    args=SFTConfig(
        output_dir="./llama-qlora",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=1e-4,           # adapters train with a higher learning rate than full fine-tuning
        max_length=2048,
        logging_steps=10,
        save_steps=100,
        model_init_kwargs={"dtype": torch.bfloat16},
    ),
)
trainer.train()
trainer.save_model()   # saves only the LoRA adapter (tens of MB)
merge_adapter.pyPYTHON
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Merge the adapter into the base model and save one model β†’ for vLLM serving or GGUF conversion
model = AutoPeftModelForCausalLM.from_pretrained("./llama-qlora", dtype=torch.bfloat16)
merged = model.merge_and_unload()
merged.save_pretrained("./llama-merged")
AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct").save_pretrained("./llama-merged")
⚠️

Run the same evaluation set before and after fine-tuning. It is common for a specific task to improve while general ability or safety drops. If you distribute the fine-tuned model, the license requires its name to start with "Llama".

The full flow β€” cleaning data, reading training logs, comparing before and after, GGUF conversion, and Ollama import β€” continues in steps 6-9 of the hands-on tutorial.

Troubleshooting

SymptomCauseFix
Responses are very slowThe model does not fit in VRAM, so part of it runs on the CPUCheck the CPU share with ollama ps β†’ pick a smaller model or quantization, shorten the context
Forgets the start of long documentsInput beyond the default context (4K, etc.) is cut offRaise OLLAMA_CONTEXT_LENGTH or num_ctx
Answers in prose instead of calling toolsSmall model, vague tool descriptions, high temperatureUse 8B or larger, make tool descriptions specific, temperature=0
JSON fails to parseThe format is only requested in the promptPass a JSON schema to format/response_format
CUDA OOM when vLLM starts--max-model-len is too large to reserve the KV cacheLower --max-model-len, adjust --gpu-memory-utilization, increase tensor parallelism
Hugging Face 401/403License not accepted for the gated model, or no tokenAccept it on the model page, then hf auth login

Frequently asked questions

Can I use Llama commercially for free?

Most services can use Llama commercially for free under the Llama Community License. Services with more than 700 million monthly active users on the model’s release date must request a separate license from Meta, and there are conditions such as displaying "Built with Llama" and following the Acceptable Use Policy, so read the license for the version you use.

Should I use Llama 4 or Llama 3.x?

On a local PC or a single consumer GPU, Llama 3.1 8B or Llama 3.2 3B is realistic. Llama 4 Scout is fast because only 17B parameters are active per token, but its full weights are 109B, so it needs that much memory. Choose Llama 4 if you need image input or very long context and have a GPU server; for a 70B-class text server, look at Llama 3.3 70B first.

How much VRAM do I need to run Llama 8B?

With 4-bit (Q4_K_M) quantization the weights are about 5 GB; add the KV cache for the context and it runs on an 8 GB GPU. At 16-bit the weights alone are about 16 GB. It also runs on CPU and system memory without a GPU, but responses are much slower.

How do Ollama and vLLM differ?

Ollama is easy to install and manage models with, which suits personal development and small internal services. vLLM uses PagedAttention and continuous batching for high throughput under many concurrent requests, which suits production API servers. Both expose an OpenAI-compatible API, so a common setup is Ollama for development and vLLM for production with the same code.

Why does Llama in Ollama forget the beginning of long documents?

Ollama runs with a default context length sized to your GPU memory, not the model’s maximum. Below 24 GiB of VRAM the default is 4K tokens, so longer input gets cut off at the start. Raise it with the OLLAMA_CONTEXT_LENGTH environment variable or the num_ctx option on the request.