Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
  1. Home
  2. Learn
  3. AI Core
  4. LLM Fundamentals
Guide to core LLM concepts

🧠 LLM Fundamentals Complete Guide

Visitors

From the history of LLMs to the Transformer architecture, tokenization, pre-training and fine-tuning, embeddings and RAG, decoding strategies, hallucination, the model landscape, evaluation, Agents, and safety β€” every core concept to know before working with LLMs, covered in a way that is easy but deep.

  • Beginner
  • Updated Oct 1, 2026
  • ~12 min read
  • 13 sections
  • 5 code examples

Learning Paths that include this guide

This guide is one step in the paths below. Study it together with the steps before and after.

  • AI Foundations β†’
Understand how LLMs workTransformer architecturePrompt engineering basicsBackground for AI Agent/RAG developmentLLM evaluation and safety

Contents

0 / 16
  1. How to use this guide
  2. Architecture diagrams
  3. History of LLMs
  4. Transformer architecture
  5. Tokenization
  6. Pre-training and fine-tuning
  7. Model size and quantization
  8. Context Window and embeddings
  9. Decoding strategies
  10. Prompt engineering basics
  11. Hallucination and limits
  12. The model landscape
  13. Evaluation and benchmarks
  14. Agents and Tool Use
  15. Safety and alignment
  16. Frequently asked questions
Contents 16 sections
  1. How to use this guide
  2. Architecture diagrams
  3. History of LLMs
  4. Transformer architecture
  5. Tokenization
  6. Pre-training and fine-tuning
  7. Model size and quantization
  8. Context Window and embeddings
  9. Decoding strategies
  10. Prompt engineering basics
  11. Hallucination and limits
  12. The model landscape
  13. Evaluation and benchmarks
  14. Agents and Tool Use
  15. Safety and alignment
  16. Frequently asked questions

How to use this guide

How to read it

Understanding LLM Fundamentals as a real-world workflow

From the history of LLMs to the Transformer architecture, tokenization, pre-training and fine-tuning, embeddings and RAG, decoding strategies, hallucination, the model landscape, evaluation, Agents, and safety β€” every core concept to know before working with LLMs, covered in a way that is easy but deep. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.

Key perspective

AI / LLM systems

Look beyond the model and the prompt: connect the data flow, evaluation, and post-deployment operating metrics in one view.

Understand how LLMs workTransformer architecturePrompt engineering basicsBackground for AI Agent/RAG developmentLLM evaluation and safety

Architecture diagrams

To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning LLM Fundamentals.

Learning flow

Rendering diagram…

Architecture view

Rendering diagram…

How LLMs got here

When you first open LLM Fundamentals, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.

Deep learning for language started with RNNs/LSTMs. They read a sentence one word at a time in order, accumulating information in a hidden state, but they forgot the beginning as sentences grew long, and sequential processing made them hard to parallelize, which limited large-scale training. The Transformer proposed in the 2017 paper "Attention Is All You Need" broke through that limit by laying the whole sentence out at once and computing the relationships between words in parallel instead of reading in order. Models such as GPT and BERT then put massive data on top of that architecture, leading to the LLMs of today.
PeriodMilestoneKey change
~2016RNN / LSTM based translation and language modelsSequential processing, weak on long-range dependencies
2017The Transformer arrives ("Attention Is All You Need")Parallel processing possible, Self-Attention captures long-range relationships
2018-2019BERT / GPT-2The large-scale pre-training + fine-tuning paradigm takes hold
2020GPT-3Few-shot ability improves sharply as parameter counts grow (scaling laws)
2022 onwardChatGPT / RLHF alignmentMoves beyond plain text completion to conversation and instruction following
2023 onwardOpen-source LLMs spread (LLaMA and others), multimodal, AgentsAnyone can run an LLM locally; expansion into tool use and agents

Tip

A "scaling law" is the empirical rule that performance improves predictably as model size, data volume, and compute grow together β€” one of the main engines of recent LLM progress.

Transformer architecture

Here you look at Transformer architecture alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Almost every LLM today is built on the Transformer architecture from the 2017 paper "Attention Is All You Need". The core is Self-Attention β€” the mechanism by which each word in a sentence computes how relevant every other word is, to take context into account. Intuitively, when reading the pronoun "it", the model scans all the other words to work out what it refers to and assigns each a relevance (weight). In implementation, each word is turned into three vectors β€” Query (what I am looking for), Key (what I have), and Value (the information actually passed on) β€” and the weights come from the similarity of Query and Key, which are then used to take a weighted sum of the Values.
attention_intuition.pyPYTHON
import torch
import torch.nn.functional as F

def simple_attention(Q, K, V):
    # Q, K, V: (seq_len, d_k)
    scores = Q @ K.T / (K.shape[-1] ** 0.5)   # similarity + scaling
    weights = F.softmax(scores, dim=-1)       # normalize relevance like probabilities
    return weights @ V                        # mix in context through a weighted sum

seq_len, d_k = 4, 8
Q = torch.randn(seq_len, d_k)
K = torch.randn(seq_len, d_k)
V = torch.randn(seq_len, d_k)
context = simple_attention(Q, K, V)
print(context.shape)   # torch.Size([4, 8]) β€” each token is updated into a context-aware vector
ComponentRole
Self-Attention (QΒ·KΒ·V)Each token computes its relevance (weight) to the other tokens in the sentence and mixes in their information
Multi-Head AttentionRuns several attentions in parallel to capture relationships from different viewpoints (grammar, meaning, and so on)
Feed-Forward LayerApplies a non-linear transform to the attention output to raise expressive power
Positional EncodingInjects word order information into the model (attention itself knows nothing about order)
Decoder-only (GPT style)Predicts the next token from the previous tokens only β€” the structure of most LLM chatbots

Tip

Most conversational LLMs, including GPT, Claude, and LLaMA, are Decoder-only Transformers rather than Encoder-Decoder β€” a structure that predicts "what is the next token" over and over.

Tokenization

Here you look at Tokenization alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

An LLM does not see characters directly; it processes text in tokens. Most LLMs build their vocabulary with a BPE (Byte Pair Encoding) family algorithm that merges frequent character combinations into a single token. For example, the word "learning" may be one token as a whole, or it may be split in two as "learn" and "ing" β€” because the vocabulary is decided statistically from frequencies in the training data.
tokenize_demo.pyPYTHON
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

text = "An AI Agent calls tools on top of an LLM"
tokens = tokenizer.tokenize(text)
ids    = tokenizer.encode(text)

print(tokens)         # split into subword units, not words
print(len(ids))       # token count β€” non-English text tends to use more tokens than English
print(tokenizer.decode(ids))   # restore the original text
ApproachUnitPros and cons
Word levelwhole wordIntuitive, but the vocabulary becomes huge and it is weak on new words
Character levelcharacterSmall vocabulary, but sentences become very long and inefficient
Subword (BPE and others)subwordThe LLM standard β€” the balance point between vocabulary size and sequence length

Tip

Languages other than English (Korean or Japanese, for example) often use more tokens to express the same meaning β€” factor that in when designing for API cost and the Context Window.

Pre-training and fine-tuning

Here you look at Pre-training and fine-tuning alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

An LLM is usually built in several stages: pre-training, which learns "next-token prediction" on vast text; supervised fine-tuning (SFT), which adjusts it to follow instructions; and then a stage that aligns it with human preferences.
lora_concept.pyPYTHON
# The core idea of LoRA: freeze the original weights W and
# train only the product of two small matrices A and B (a low-rank matrix), which is added to W.
#   W_new = W_frozen + (A @ B) * scale
#
# If the original weights are 4096x4096 = 16 million parameters,
# LoRA at rank r=8 trains only 4096x8 + 8x4096 = about 65 thousand parameters
# -> hundreds of times less memory and storage than full fine-tuning

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)
model.print_trainable_parameters()   # check the share of trainable parameters
StagePurposeData scale
Pre-trainingAcquire the general patterns and knowledge of languageHundreds of billions to trillions of tokens
Supervised fine-tuning (SFT)Adjust the response style to follow instructionsTens to hundreds of thousands of examples
RLHF (reinforcement learning from human feedback)Reinforcement learning toward the answers people preferPreference rankings assigned by people
DPO (direct preference optimization)Reaches a goal similar to RLHF more simply, without a separate reward modelPreference ranking data (lighter to train than RLHF)
Lightweight fine-tuning (LoRA / QLoRA)Trains only a small number of extra parameters instead of all the weights, specializing with few resourcesHundreds to thousands of examples

Tip

To the question "should I fine-tune the model or solve it with prompting", the cost-effective answer is usually to try prompting/RAG first and consider fine-tuning only for recurring style and format problems that still do not go away.

Model size and quantization

Model size and quantization is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

Performance generally rises with model size (parameter count), but so do memory and compute costs. Quantization is a technique that lowers the precision of the weights (for example 32-bit β†’ 4-bit) to cut memory use, which makes it possible to run large models even on a single laptop.
Format/techniqueDescription
FP16 / BF16The 16-bit floating point commonly used for training and inference (half the original precision)
INT8 / INT4 quantizationApproximates weights as 8-bit or 4-bit integers, cutting memory by a further half to three quarters
GGUFThe quantized model file format used by the llama.cpp family β€” widely used for local CPU/GPU inference
GPTQ / AWQThe main algorithms for quantizing to the 4-bit level with minimal accuracy loss

Tip

  • Parameter count (7B, 70B, and so on) is only a rough indicator of model performance; depending on training data quality and alignment, a smaller model is often the more practical choice.
  • Quantization is a tradeoff that gives up precision for speed and memory efficiency β€” it matters most when running LLMs in local/edge environments.

Context Window and embeddings

Context Window and embeddings is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

The Context Window is the maximum number of tokens a model can refer to at once. To work with knowledge beyond that limit, you use RAG (Retrieval-Augmented Generation): embedding-based search (turning text into vectors) selects only the relevant content to put into the context. Instead of making the model memorize everything, RAG is like an open-book exam, where the reference material is found and handed over when needed.
ConceptDescription
Context WindowThe total tokens a model can refer to in one request (prompt + response)
EmbeddingText converted into a fixed-length vector that reflects its meaning
Cosine similarityComputes semantic similarity from how closely two embedding vectors point in the same direction
Vector searchFinds the document embeddings closest to the question embedding to pick out relevant documents
RAGAdds the documents found by vector search to the prompt to supply fresh or specialist knowledge

Tip

A bigger Context Window is not automatically better β€” when a lot of irrelevant content is mixed in, answer quality can actually drop, a reported effect called "lost in the middle". Embeddings and vector search are covered in more depth in the vectordb, langchain, and llamaindex guides.

Decoding strategies: why the same model speaks differently

Here you look at Decoding strategies: why the same model speaks differently alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

At each step an LLM only outputs "a probability distribution over the next token"; which token is actually picked is decided by the decoding strategy. Even with the same model, these settings change the consistency and creativity of the answers considerably.
decoding_params.pyPYTHON
from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a short poem"}],
    temperature=0.9,   # higher means more creative and varied wording
    top_p=0.95,        # sample only within the top 95% of probability mass
)
print(resp.choices[0].message.content)
Strategy/parameterDescriptionEffect
Greedy DecodingPicks only the highest-probability token at every stepConsistent but repetitive and flat answers
TemperatureControls how flat or peaked the probability distribution isLow is deterministic, high is varied and random
Top-kSamples only from the top k tokens by probabilityRules out strange tokens with very low probability
Top-p (nucleus)Samples from the candidate set whose cumulative probability reaches pThe size of the candidate set adapts flexibly to the situation
Beam SearchKeeps several candidate paths at once and searches for the most plausible sentence overallGood for tasks with a fairly clear right answer, such as translation

Tip

A common starting point is a low temperature of about 0-0.3 for fact-based answers (code generation, data extraction) and a higher 0.7-1.0 for creative writing.

Prompt engineering basics

Here you look at Prompt engineering basics alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Even with the same model, result quality varies widely with how the prompt is put together. Below are the most basic patterns.
prompt_patterns.pyPYTHON
SYSTEM_PROMPT = "You are an assistant that answers only with accurate facts. If you do not know, say so."

FEW_SHOT_EXAMPLES = """
Question: What is 2 + 2?
Answer: 4

Question: Which planet is closest to Earth?
Answer: Venus.
"""

def build_prompt(question: str) -> str:
    return f"{FEW_SHOT_EXAMPLES}\nQuestion: {question}\nAnswer:"
PatternDescriptionExample
Zero-shotAsk directly with no examples"Classify the sentiment of this review: ..."
Few-shotProvide a few input-output examples along with the question2-3 examples + the new question
Chain-of-Thought (CoT)Lead the model to work through its thinking step by step before answering"Think step by step"
Role assignment (System Prompt)Fix the role and constraints of the model at the start of the conversation"You are a code reviewer..."
ReAct (reasoning + acting)Alternates thinking (Reasoning) and tool calls (Acting)The pattern where an Agent keeps reasoning while using a search tool

Tip

Adding the single sentence "Think step by step" often raises accuracy noticeably β€” the power of Chain-of-Thought prompting.

Hallucination and the limits of LLMs

Hallucination and the limits of LLMs is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

An LLM does not "look up" facts; it "generates" the most plausible next token based on the patterns it learned. That is why hallucination β€” confidently making up papers, APIs, or facts that do not exist as if they were real β€” can happen structurally.
LimitDescriptionMitigation
HallucinationGenerates content that sounds plausible but is not trueSupply grounding documents with RAG, require source citations, low temperature
No recent informationKnows nothing after the point its training data was collected (knowledge cutoff)Add real-time information with a web search tool or RAG
Weak at numbers and arithmeticWeak at exact calculation with large numbers and precise countingDelegate to a calculator or code execution tool (Tool Use)
InconsistencyThe same question phrased differently can get a different answerLower the temperature, or sample several times and take a majority vote

Tip

Design your system with the attitude that "an LLM answer is always a draft that needs verification" β€” especially in domains where a wrong answer is costly, such as medicine, law, and finance, always enforce source citations or add a final human check.

The model landscape: what to choose when

The model landscape: what to choose when is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

The LLM ecosystem divides broadly into closed models, offered only through an API, and open-weight models, whose weights you can download and run yourself. Each has clear pros and cons, so choose according to the requirements of the project.
CategoryMain modelsTraits
Closed APIGPT (OpenAI), Claude (Anthropic), Gemini (Google)Top-tier performance, no infrastructure to manage, data is sent outside
Open source/open weightLLaMA (Meta), Mistral, Qwen, GemmaCan be self-hosted, high freedom to fine-tune, you carry the infrastructure and operations burden

Tip

  • If data sensitivity is high (internal documents, personal data) or costs must be kept predictable, look at open-source self-hosting first; if top performance and fast development come first, look at API-based models first.
  • Loading a real model and running inference in the huggingface guide, then moving on to the langchain and llamaindex guides, follows naturally from here.

LLM evaluation and benchmarks

LLM evaluation and benchmarks is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

Saying "this model is better than that one" needs a yardstick. LLM evaluation divides broadly into standardized benchmark scores and custom evaluation matched to the real task.
Benchmark/methodWhat it measures
MMLUBroad knowledge and reasoning through multiple-choice questions across 57 subjects
HumanEvalThe ability to write working code from a function description
HellaSwagCommon-sense reasoning by choosing the sentence that follows naturally from the context
LLM-as-a-JudgeHaving another strong LLM grade the quality of the responses
Human evaluation (Human Eval)Real users and experts compare and rate the responses directly

Tip

A high rank on a general benchmark does not guarantee the model works well in your service β€” building a small evaluation set (golden set) that reflects your real scenarios and regression-testing against it regularly is the most reliable method.

Agents and Tool Use: when the LLM starts to act

Agents and Tool Use: when the LLM starts to act is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

A plain LLM only exchanges text, but combined with function/tool calling it becomes an Agent that decides for itself to perform external actions such as search, calculation, and DB lookups. The ReAct pattern seen earlier (repeating thought β†’ action β†’ observation) is the basic loop of that Agent behavior.
ConceptDescription
Function/Tool CallingThe model reads a schema saying "call this tool in this situation" and decides by itself which tool to call with which arguments
ReAct loopApproaches the goal by repeating thought (Reasoning) β†’ tool call (Acting) β†’ observing the result (Observation)
Multi-agentSeveral Agents with different roles collaborate on or review one task
MCP (Model Context Protocol)An open protocol that lets models connect to external tools and data sources in a standardized way

Tip

The concrete code for designing and implementing Agents continues in the langgraph, langchain, and huggingface guides β€” in this guide it is enough to grasp the concept of "why this structure is needed".

Safety and alignment

Safety and alignment is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.

The more powerful the model, the greater the potential for harmful answers, biased judgments, and misuse. Alignment refers to the whole process of matching the behavior of a model to human intent and values.
ConceptDescription
RLHFAligns the model through reinforcement learning, using human preference rankings as the reward signal
Constitutional AITrains the model to critique and revise its own answers against a set of principles (a constitution), reducing human involvement
Red TeamingDeliberately provoking weaknesses and harmful responses to find and fix them in advance
GuardrailsApplication-level safeguards such as input/output filtering and blocking forbidden topics

Tip

  • Alignment is not done once training ends; it is an ongoing process in which new bypass attempts (jailbreaks) keep being found β€” when you put an LLM into a service, do not rely only on the safeguards of the model itself but add application-level guardrails as well.
  • That completes the basic fitness for LLMs. Now move on to the huggingface, langchain, llamaindex, and langgraph guides to actually load models and build Agents/RAG.

LLM Fundamentals: frequently asked questions

What is an LLM?

An LLM (large language model) is trained on a huge amount of text and generates sentences by predicting the token that comes next. Answering questions, summarizing, and writing code are all the result of repeating that next-token prediction. That is why a model can produce text that sounds right but is wrong, and why information newer than its training data, or internal documents, has to be supplied separately.

What are tokens and the context window?

A token is the unit a model processes text in, often a piece smaller than a word. The context window is the maximum number of tokens the model can see in one request, and it includes the prompt, the conversation history, and the answer being generated. The same content can take a different number of tokens depending on the language, so measure cost and length with actual token counts.

Should I choose RAG or fine-tuning?

To make the model answer from internal documents or recent information it does not know, RAG is the fit. When documents change you only re-index, and the answer can show its sources. Fine-tuning is for consistently changing the format or tone of answers, or the way a specific task is carried out. Most teams start with better prompts and RAG, then add fine-tuning for what is still missing.

Why do hallucinations happen and how do I reduce them?

A model does not look facts up; it generates a plausible next token, so it writes fluent sentences even without evidence. To reduce this, put the supporting documents in the prompt (RAG), instruct the model to say it does not know when there is no evidence, and have it output sources so the answer can be checked. Hallucinations cannot be removed completely, so measure important outputs with an evaluation set and keep a human review step.

← Previous guideML FundamentalsNext guide β†’Python AI