Run, fine-tune, and serve LLMs locally with Hugging Face Transformers, PEFT, and TGI. Covers LoRA/QLoRA fine-tuning, the Inference API, a Text Generation Inference production server, and agent integration.
Intermediate
Updated Oct 1, 2026
~5 min read
11 sections
9 code examples
Web IDE lab included
HF
Hugging Face Web IDE
Run code in the browser with nothing to install and learn through step-by-step examples.
Understanding Hugging Face as a real-world workflow
Run, fine-tune, and serve LLMs locally with Hugging Face Transformers, PEFT, and TGI. Covers LoRA/QLoRA fine-tuning, the Inference API, a Text Generation Inference production server, and agent integration. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
AI / LLM systems
Look beyond the model and the prompt: connect the data flow, evaluation, and post-deployment operating metrics in one view.
LLM fine-tuning (LoRA/QLoRA)TGI production serverInference APIManaging PEFT adapters
Architecture diagrams
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Hugging Face.
Learning flow
Rendering diagramβ¦
Architecture view
Rendering diagramβ¦
Model hub & pipelines
When you first open Hugging Face, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
More than 500,000 models are published on the Hugging Face Hub. The pipeline API abstracts model loading, tokenizing, inference, and post-processing into a single line, enabling fast prototyping. In production, AutoTokenizer + AutoModelForCausalLM give you fine-grained control.
Unified API for loading models and running inference
datasets
Loading training datasets and streaming preprocessing
peft
LoRA/QLoRA lightweight fine-tuning adapters
trl
Alignment training trainers such as SFT, DPO, and PPO
accelerate
Launcher for multi-GPU and distributed training
huggingface_hub
Model upload, download, and versioning
Inference API
Here you look at Inference API alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
The Inference API calls hundreds of models as HTTP endpoints right away, with no GPU. InferenceClient supports streaming by default and comes in two plans: Serverless (free) and Dedicated Endpoint (a dedicated GPU). Use a Dedicated Endpoint when you need a production SLA.
PYTHON
from huggingface_hub import InferenceClientclient = InferenceClient(api_key="hf_...")# Streaming response β live token outputresponse = client.chat_completion( model="meta-llama/Llama-3.1-70B-Instruct", messages=[{"role": "user", "content": "Explain the key concepts of LLM fine-tuning"}], max_tokens=1024, stream=True,)for chunk in response: print(chunk.choices[0].delta.content, end="", flush=True)
LoRA fine-tuning
Here you look at LoRA fine-tuning alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
LoRA (Low-Rank Adaptation) freezes the original weights and trains only a pair of low-rank matrices (AΒ·B). It reaches domain-specific performance with trainable parameters under 0.1% of the total, which has made it the de facto standard for custom LLM development. For target_modules, choose q_proj/v_proj or c_attn depending on the model architecture.
PYTHON
from peft import LoraConfig, get_peft_model, TaskTypefrom transformers import AutoModelForCausalLMimport torch# LoRA config β a higher r (rank) means more capacity and more memorylora_config = LoraConfig( r=16, lora_alpha=32, # scaling factor (usually 2x r) target_modules=["q_proj", "v_proj"], # train only the attention query and value layers lora_dropout=0.05, bias="none", task_type=TaskType.CAUSAL_LM,)model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-3.1-8B", torch_dtype=torch.bfloat16, device_map="auto",)model = get_peft_model(model, lora_config)model.print_trainable_parameters()# trainable params: 6,815,744 || all params: 8,036,966,400 || trainable%: 0.08%
QLoRA (4-bit quantization)
Here you look at QLoRA (4-bit quantization) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
QLoRA combines LoRA with 4-bit NF4 quantization, cutting VRAM use by more than 70%. That makes fine-tuning a 70B model possible on a single 24GB GPU. double_quant quantizes the quantization constants again for extra savings.
PYTHON
from transformers import BitsAndBytesConfig, AutoModelForCausalLMfrom peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_trainingimport torch# 4-bit NF4 quantization configbnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", # NF4 performs better than fp4 bnb_4bit_compute_dtype=torch.bfloat16, # computation is done in bfloat16 bnb_4bit_use_double_quant=True, # extra savings from double quantization)model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-3.1-8B", quantization_config=bnb_config, device_map="auto",)# Enable gradient checkpointing for 4-bit trainingmodel = prepare_model_for_kbit_training(model)lora_config = LoraConfig(r=64, lora_alpha=16, task_type=TaskType.CAUSAL_LM)model = get_peft_model(model, lora_config)
Preparing datasets
Here you look at Preparing datasets alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
80% of fine-tuning quality depends on data quality. The datasets library supports streaming, sharding, and parallel preprocessing, so it can handle even billions of tokens. apply_chat_template applies the official chat format of each model automatically, preventing a mismatch between training and inference.
PYTHON
from datasets import load_datasetfrom transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")# Load a JSONL file β in the form {"input": ..., "output": ...}dataset = load_dataset("json", data_files="train.jsonl", split="train")def format_chat(example): messages = [ {"role": "system", "content": "You are a friendly AI assistant."}, {"role": "user", "content": example["input"]}, {"role": "assistant", "content": example["output"]}, ] # Apply the chat template of the model (returns a string without tokenizing) return {"text": tokenizer.apply_chat_template(messages, tokenize=False)}dataset = dataset.map(format_chat, remove_columns=dataset.column_names)print(dataset[0]["text"][:300])
TGI production server
Here you look at TGI production server alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Text Generation Inference (TGI) is the high-performance LLM serving engine built by HuggingFace. With continuous batching, PagedAttention, and tensor parallelism built in, it delivers throughput on par with vLLM. It supports an OpenAI-compatible endpoint (/v1/chat/completions), so existing code connects without modification.
BASH
# Run the TGI Docker server β single GPUdocker run --gpus all -p 8080:80 \ -v $HOME/.cache/huggingface:/data \ ghcr.io/huggingface/text-generation-inference:3.0 \ --model-id meta-llama/Llama-3.1-8B-Instruct \ --max-total-tokens 4096 \ --max-input-tokens 3072
PYTHON
# Call TGI with an OpenAI-compatible clientfrom openai import OpenAIclient = OpenAI(base_url="http://localhost:8080/v1", api_key="none")response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "What is the weather in Seoul?"}], stream=True, max_tokens=512,)for chunk in response: print(chunk.choices[0].delta.content or "", end="", flush=True)
Model evaluation
Here you look at Model evaluation alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Objective quality measurement is essential after fine-tuning. The evaluate library offers BLEU, ROUGE, and BERTScore through a unified interface. ROUGE-L suits summarization and BERTScore suits semantic similarity, and for domain-specific benchmarks it is best to build a separate evaluation set.
PYTHON
import evaluate# Load the metrics (downloaded automatically on first run)rouge = evaluate.load("rouge")bertscore = evaluate.load("bertscore")predictions = ["This is the summary generated by the model."]references = ["This is the reference summary."]# ROUGE β based on word overlap (fast to compute)rouge_result = rouge.compute(predictions=predictions, references=references)print(rouge_result)# {'rouge1': 0.82, 'rouge2': 0.71, 'rougeL': 0.79}# BERTScore β based on semantic embeddings (more accurate, GPU recommended)bert_result = bertscore.compute( predictions=predictions, references=references, lang="en")print(f"F1: {bert_result['f1'][0]:.4f}")
HF Agents & Tool
Here you look at HF Agents & Tool alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
smolagents is the lightweight agent library from HuggingFace. A CodeAgent calls tools by having the LLM write and run Python code directly, which handles more complex logic than JSON-based tool calls. HfApiModel uses the Inference API as the brain of the agent with no separate infrastructure.
PYTHON
from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel# Use the Inference API as the agent LLMmodel = HfApiModel(model_id="meta-llama/Llama-3.1-70B-Instruct")agent = CodeAgent( tools=[DuckDuckGoSearchTool()], # web search tool model=model, max_steps=10,)# The agent searches β analyzes β generates code β returns the resultresult = agent.run("Find the latest LLM benchmark results and summarize them in a table")print(result)
Hugging Face practical design
Hugging Face practical design is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
With Hugging Face, get the management of the tokenizer, config, pipeline, and fine-tuning artifacts right before choosing a model. Managing the model card and dataset card together improves reproducibility.
Decision point
Question to ask
Practical guideline
Boundaries
Which parts of the Hugging Face code are most likely to change?
Separate input/output, configuration, external integrations, and the core rules.
State
Where is state created and where does it go away?
Make the owner and lifecycle of state visible in the code.
Failure
What does the caller receive on failure?
Decide the timeout, fallback, and error contract first.
Hugging Face operating standards
This section covers Hugging Face operating standards from a practical perspective. Instead of memorizing the concept, focus on the situations in which you would reach for it.
In inference operations, max tokens, batch size, quantization, device_map, and the cache policy determine cost and latency. For large models, treat load time and peak memory as separate metrics too.
Hugging Face verification strategy
Hugging Face verification strategy is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
Pin a benchmark set, safety samples, and regression prompts and compare them before and after prompt/dataset changes.
Quality axis
How to verify
Definition of done
Correctness
Automate the normal/failure cases.
The core scenarios pass reproducibly.
Regression prevention
When fixing a bug, leave the same case behind as a test.
The same failure is not deployed again.
Operability
Check the logs, metrics, and alerts.
There is a path for tracing the cause when a problem occurs.
Hugging Face: frequently asked questions
What does Hugging Face provide?
It provides the Hub, where public models and datasets are uploaded and downloaded, and libraries such as transformers and datasets that load those models for inference and training. Naming a model is enough to fetch its weights and tokenizer, which has made it the de facto standard path for working with open models.
Can I use models from the Hub commercially?
Licenses differ per model, so always check the license on the model card. Some licenses allow free use, while other models are limited to non-commercial use or require accepting terms before download. A fine-tuned model usually inherits the license terms of its base model.
How do I manage where models are downloaded and how much space they take?
Downloaded files go to a cache directory in the user's home, and the HF_HOME environment variable changes the location. Large models can be tens of gigabytes, so point the cache at a path with plenty of disk and remove unused models with the cache management command. On servers and in containers, download at build time or keep the cache on a volume so models are not fetched again every time.