Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
  1. Home
  2. Learn
  3. AI Applied Development
  4. Hugging Face
Guide to the Hugging Face model hub & fine-tuning

HF Hugging Face Complete Guide

Visitors

Run, fine-tune, and serve LLMs locally with Hugging Face Transformers, PEFT, and TGI. Covers LoRA/QLoRA fine-tuning, the Inference API, a Text Generation Inference production server, and agent integration.

  • Intermediate
  • Updated Oct 1, 2026
  • ~5 min read
  • 11 sections
  • 9 code examples
  • Web IDE lab included
HF

Hugging Face Web IDE

Run code in the browser with nothing to install and learn through step-by-step examples.

Open the Web IDE β†’
LLM fine-tuning (LoRA/QLoRA)TGI production serverInference APIManaging PEFT adapters

Related frameworks & environments

🐍Python AIβ†’πŸ”₯PyTorchβ†’TFTensorFlowβ†’LCLangChainβ†’LILlamaIndexβ†’

Contents

0 / 14
  1. How to use this guide
  2. Architecture diagrams
  3. Model hub & pipelines
  4. Inference API
  5. LoRA fine-tuning
  6. QLoRA (4-bit)
  7. Preparing datasets
  8. TGI production server
  9. Model evaluation
  10. HF Agents & Tool
  11. Hugging Face design
  12. Operating standards
  13. Verification strategy
  14. Frequently asked questions
Contents 14 sections
  1. How to use this guide
  2. Architecture diagrams
  3. Model hub & pipelines
  4. Inference API
  5. LoRA fine-tuning
  6. QLoRA (4-bit)
  7. Preparing datasets
  8. TGI production server
  9. Model evaluation
  10. HF Agents & Tool
  11. Hugging Face design
  12. Operating standards
  13. Verification strategy
  14. Frequently asked questions

How to use this guide

How to read it

Understanding Hugging Face as a real-world workflow

Run, fine-tune, and serve LLMs locally with Hugging Face Transformers, PEFT, and TGI. Covers LoRA/QLoRA fine-tuning, the Inference API, a Text Generation Inference production server, and agent integration. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.

Key perspective

AI / LLM systems

Look beyond the model and the prompt: connect the data flow, evaluation, and post-deployment operating metrics in one view.

LLM fine-tuning (LoRA/QLoRA)TGI production serverInference APIManaging PEFT adapters

Architecture diagrams

To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Hugging Face.

Learning flow

Rendering diagram…

Architecture view

Rendering diagram…

Model hub & pipelines

When you first open Hugging Face, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.

More than 500,000 models are published on the Hugging Face Hub. The pipeline API abstracts model loading, tokenizing, inference, and post-processing into a single line, enabling fast prototyping. In production, AutoTokenizer + AutoModelForCausalLM give you fine-grained control.
PYTHON
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
import torch

# Text generation pipeline β€” device_map="auto" assigns GPU/CPU automatically
pipe = pipeline(
    "text-generation",
    model="meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Implement quicksort in Python"}]
result = pipe(messages, max_new_tokens=512, temperature=0.7)
print(result[0]["generated_text"][-1]["content"])
LibraryRole
transformersUnified API for loading models and running inference
datasetsLoading training datasets and streaming preprocessing
peftLoRA/QLoRA lightweight fine-tuning adapters
trlAlignment training trainers such as SFT, DPO, and PPO
accelerateLauncher for multi-GPU and distributed training
huggingface_hubModel upload, download, and versioning

Inference API

Here you look at Inference API alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

The Inference API calls hundreds of models as HTTP endpoints right away, with no GPU. InferenceClient supports streaming by default and comes in two plans: Serverless (free) and Dedicated Endpoint (a dedicated GPU). Use a Dedicated Endpoint when you need a production SLA.
PYTHON
from huggingface_hub import InferenceClient

client = InferenceClient(api_key="hf_...")

# Streaming response β€” live token output
response = client.chat_completion(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[{"role": "user", "content": "Explain the key concepts of LLM fine-tuning"}],
    max_tokens=1024,
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content, end="", flush=True)

LoRA fine-tuning

Here you look at LoRA fine-tuning alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

LoRA (Low-Rank Adaptation) freezes the original weights and trains only a pair of low-rank matrices (AΒ·B). It reaches domain-specific performance with trainable parameters under 0.1% of the total, which has made it the de facto standard for custom LLM development. For target_modules, choose q_proj/v_proj or c_attn depending on the model architecture.
PYTHON
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM
import torch

# LoRA config β€” a higher r (rank) means more capacity and more memory
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,                          # scaling factor (usually 2x r)
    target_modules=["q_proj", "v_proj"],    # train only the attention query and value layers
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 8,036,966,400 || trainable%: 0.08%

QLoRA (4-bit quantization)

Here you look at QLoRA (4-bit quantization) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

QLoRA combines LoRA with 4-bit NF4 quantization, cutting VRAM use by more than 70%. That makes fine-tuning a 70B model possible on a single 24GB GPU. double_quant quantizes the quantization constants again for extra savings.
PYTHON
from transformers import BitsAndBytesConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training
import torch

# 4-bit NF4 quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NF4 performs better than fp4
    bnb_4bit_compute_dtype=torch.bfloat16,  # computation is done in bfloat16
    bnb_4bit_use_double_quant=True,         # extra savings from double quantization
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
)
# Enable gradient checkpointing for 4-bit training
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(r=64, lora_alpha=16, task_type=TaskType.CAUSAL_LM)
model = get_peft_model(model, lora_config)

Preparing datasets

Here you look at Preparing datasets alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

80% of fine-tuning quality depends on data quality. The datasets library supports streaming, sharding, and parallel preprocessing, so it can handle even billions of tokens. apply_chat_template applies the official chat format of each model automatically, preventing a mismatch between training and inference.
PYTHON
from datasets import load_dataset
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# Load a JSONL file β€” in the form {"input": ..., "output": ...}
dataset = load_dataset("json", data_files="train.jsonl", split="train")

def format_chat(example):
    messages = [
        {"role": "system",    "content": "You are a friendly AI assistant."},
        {"role": "user",      "content": example["input"]},
        {"role": "assistant", "content": example["output"]},
    ]
    # Apply the chat template of the model (returns a string without tokenizing)
    return {"text": tokenizer.apply_chat_template(messages, tokenize=False)}

dataset = dataset.map(format_chat, remove_columns=dataset.column_names)
print(dataset[0]["text"][:300])

TGI production server

Here you look at TGI production server alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Text Generation Inference (TGI) is the high-performance LLM serving engine built by HuggingFace. With continuous batching, PagedAttention, and tensor parallelism built in, it delivers throughput on par with vLLM. It supports an OpenAI-compatible endpoint (/v1/chat/completions), so existing code connects without modification.
BASH
# Run the TGI Docker server β€” single GPU
docker run --gpus all -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:3.0 \
  --model-id meta-llama/Llama-3.1-8B-Instruct \
  --max-total-tokens 4096 \
  --max-input-tokens 3072
PYTHON
# Call TGI with an OpenAI-compatible client
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "What is the weather in Seoul?"}],
    stream=True,
    max_tokens=512,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Model evaluation

Here you look at Model evaluation alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Objective quality measurement is essential after fine-tuning. The evaluate library offers BLEU, ROUGE, and BERTScore through a unified interface. ROUGE-L suits summarization and BERTScore suits semantic similarity, and for domain-specific benchmarks it is best to build a separate evaluation set.
PYTHON
import evaluate

# Load the metrics (downloaded automatically on first run)
rouge     = evaluate.load("rouge")
bertscore = evaluate.load("bertscore")

predictions = ["This is the summary generated by the model."]
references  = ["This is the reference summary."]

# ROUGE β€” based on word overlap (fast to compute)
rouge_result = rouge.compute(predictions=predictions, references=references)
print(rouge_result)
# {'rouge1': 0.82, 'rouge2': 0.71, 'rougeL': 0.79}

# BERTScore β€” based on semantic embeddings (more accurate, GPU recommended)
bert_result = bertscore.compute(
    predictions=predictions, references=references, lang="en"
)
print(f"F1: {bert_result['f1'][0]:.4f}")

HF Agents & Tool

Here you look at HF Agents & Tool alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

smolagents is the lightweight agent library from HuggingFace. A CodeAgent calls tools by having the LLM write and run Python code directly, which handles more complex logic than JSON-based tool calls. HfApiModel uses the Inference API as the brain of the agent with no separate infrastructure.
PYTHON
from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel

# Use the Inference API as the agent LLM
model = HfApiModel(model_id="meta-llama/Llama-3.1-70B-Instruct")

agent = CodeAgent(
    tools=[DuckDuckGoSearchTool()],  # web search tool
    model=model,
    max_steps=10,
)

# The agent searches β†’ analyzes β†’ generates code β†’ returns the result
result = agent.run("Find the latest LLM benchmark results and summarize them in a table")
print(result)

Tip

  • Run a CodeAgent in a sandboxed environment β€” it can execute arbitrary code.
  • Custom tools are defined with the @tool decorator, and type hints and a docstring are required.

Hugging Face practical design

Hugging Face practical design is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

With Hugging Face, get the management of the tokenizer, config, pipeline, and fine-tuning artifacts right before choosing a model. Managing the model card and dataset card together improves reproducibility.
Decision pointQuestion to askPractical guideline
BoundariesWhich parts of the Hugging Face code are most likely to change?Separate input/output, configuration, external integrations, and the core rules.
StateWhere is state created and where does it go away?Make the owner and lifecycle of state visible in the code.
FailureWhat does the caller receive on failure?Decide the timeout, fallback, and error contract first.

Hugging Face operating standards

This section covers Hugging Face operating standards from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.

In inference operations, max tokens, batch size, quantization, device_map, and the cache policy determine cost and latency. For large models, treat load time and peak memory as separate metrics too.

Tip

  • model card
  • tokenizer compatibility
  • quantization benchmark
  • safety regression prompts

Hugging Face verification strategy

Hugging Face verification strategy is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

Pin a benchmark set, safety samples, and regression prompts and compare them before and after prompt/dataset changes.
Quality axisHow to verifyDefinition of done
CorrectnessAutomate the normal/failure cases.The core scenarios pass reproducibly.
Regression preventionWhen fixing a bug, leave the same case behind as a test.The same failure is not deployed again.
OperabilityCheck the logs, metrics, and alerts.There is a path for tracing the cause when a problem occurs.

Hugging Face: frequently asked questions

What does Hugging Face provide?

It provides the Hub, where public models and datasets are uploaded and downloaded, and libraries such as transformers and datasets that load those models for inference and training. Naming a model is enough to fetch its weights and tokenizer, which has made it the de facto standard path for working with open models.

Can I use models from the Hub commercially?

Licenses differ per model, so always check the license on the model card. Some licenses allow free use, while other models are limited to non-commercial use or require accepting terms before download. A fine-tuned model usually inherits the license terms of its base model.

How do I manage where models are downloaded and how much space they take?

Downloaded files go to a cache directory in the user's home, and the HF_HOME environment variable changes the location. Large models can be tens of gigabytes, so point the cache at a path with plenty of disk and remove unused models with the cache management command. On servers and in containers, download at build time or keep the cache on a volume so models are not fetched again every time.

← Previous guideApplied AI Intro & RoadmapNext guide β†’LangChain