From the history of LLMs to the Transformer architecture, tokenization, pre-training and fine-tuning, embeddings and RAG, decoding strategies, hallucination, the model landscape, evaluation, Agents, and safety β every core concept to know before working with LLMs, covered in a way that is easy but deep.
Beginner
Updated Oct 1, 2026
~12 min read
13 sections
5 code examples
Learning Paths that include this guide
This guide is one step in the paths below. Study it together with the steps before and after.
Understanding LLM Fundamentals as a real-world workflow
From the history of LLMs to the Transformer architecture, tokenization, pre-training and fine-tuning, embeddings and RAG, decoding strategies, hallucination, the model landscape, evaluation, Agents, and safety β every core concept to know before working with LLMs, covered in a way that is easy but deep. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
AI / LLM systems
Look beyond the model and the prompt: connect the data flow, evaluation, and post-deployment operating metrics in one view.
Understand how LLMs workTransformer architecturePrompt engineering basicsBackground for AI Agent/RAG developmentLLM evaluation and safety
Architecture diagrams
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning LLM Fundamentals.
Learning flow
Rendering diagramβ¦
Architecture view
Rendering diagramβ¦
How LLMs got here
When you first open LLM Fundamentals, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
Deep learning for language started with RNNs/LSTMs. They read a sentence one word at a time in order, accumulating information in a hidden state, but they forgot the beginning as sentences grew long, and sequential processing made them hard to parallelize, which limited large-scale training. The Transformer proposed in the 2017 paper "Attention Is All You Need" broke through that limit by laying the whole sentence out at once and computing the relationships between words in parallel instead of reading in order. Models such as GPT and BERT then put massive data on top of that architecture, leading to the LLMs of today.
Period
Milestone
Key change
~2016
RNN / LSTM based translation and language models
Sequential processing, weak on long-range dependencies
2017
The Transformer arrives ("Attention Is All You Need")
The large-scale pre-training + fine-tuning paradigm takes hold
2020
GPT-3
Few-shot ability improves sharply as parameter counts grow (scaling laws)
2022 onward
ChatGPT / RLHF alignment
Moves beyond plain text completion to conversation and instruction following
2023 onward
Open-source LLMs spread (LLaMA and others), multimodal, Agents
Anyone can run an LLM locally; expansion into tool use and agents
Transformer architecture
Here you look at Transformer architecture alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Almost every LLM today is built on the Transformer architecture from the 2017 paper "Attention Is All You Need". The core is Self-Attention β the mechanism by which each word in a sentence computes how relevant every other word is, to take context into account. Intuitively, when reading the pronoun "it", the model scans all the other words to work out what it refers to and assigns each a relevance (weight). In implementation, each word is turned into three vectors β Query (what I am looking for), Key (what I have), and Value (the information actually passed on) β and the weights come from the similarity of Query and Key, which are then used to take a weighted sum of the Values.
attention_intuition.pyPYTHON
import torchimport torch.nn.functional as Fdef simple_attention(Q, K, V): # Q, K, V: (seq_len, d_k) scores = Q @ K.T / (K.shape[-1] ** 0.5) # similarity + scaling weights = F.softmax(scores, dim=-1) # normalize relevance like probabilities return weights @ V # mix in context through a weighted sumseq_len, d_k = 4, 8Q = torch.randn(seq_len, d_k)K = torch.randn(seq_len, d_k)V = torch.randn(seq_len, d_k)context = simple_attention(Q, K, V)print(context.shape) # torch.Size([4, 8]) β each token is updated into a context-aware vector
Component
Role
Self-Attention (QΒ·KΒ·V)
Each token computes its relevance (weight) to the other tokens in the sentence and mixes in their information
Multi-Head Attention
Runs several attentions in parallel to capture relationships from different viewpoints (grammar, meaning, and so on)
Feed-Forward Layer
Applies a non-linear transform to the attention output to raise expressive power
Positional Encoding
Injects word order information into the model (attention itself knows nothing about order)
Decoder-only (GPT style)
Predicts the next token from the previous tokens only β the structure of most LLM chatbots
Tokenization
Here you look at Tokenization alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
An LLM does not see characters directly; it processes text in tokens. Most LLMs build their vocabulary with a BPE (Byte Pair Encoding) family algorithm that merges frequent character combinations into a single token. For example, the word "learning" may be one token as a whole, or it may be split in two as "learn" and "ing" β because the vocabulary is decided statistically from frequencies in the training data.
tokenize_demo.pyPYTHON
from transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("gpt2")text = "An AI Agent calls tools on top of an LLM"tokens = tokenizer.tokenize(text)ids = tokenizer.encode(text)print(tokens) # split into subword units, not wordsprint(len(ids)) # token count β non-English text tends to use more tokens than Englishprint(tokenizer.decode(ids)) # restore the original text
Approach
Unit
Pros and cons
Word level
whole word
Intuitive, but the vocabulary becomes huge and it is weak on new words
Character level
character
Small vocabulary, but sentences become very long and inefficient
Subword (BPE and others)
subword
The LLM standard β the balance point between vocabulary size and sequence length
Pre-training and fine-tuning
Here you look at Pre-training and fine-tuning alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
An LLM is usually built in several stages: pre-training, which learns "next-token prediction" on vast text; supervised fine-tuning (SFT), which adjusts it to follow instructions; and then a stage that aligns it with human preferences.
lora_concept.pyPYTHON
# The core idea of LoRA: freeze the original weights W and# train only the product of two small matrices A and B (a low-rank matrix), which is added to W.# W_new = W_frozen + (A @ B) * scale## If the original weights are 4096x4096 = 16 million parameters,# LoRA at rank r=8 trains only 4096x8 + 8x4096 = about 65 thousand parameters# -> hundreds of times less memory and storage than full fine-tuningfrom peft import LoraConfig, get_peft_modelconfig = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])model = get_peft_model(base_model, config)model.print_trainable_parameters() # check the share of trainable parameters
Stage
Purpose
Data scale
Pre-training
Acquire the general patterns and knowledge of language
Hundreds of billions to trillions of tokens
Supervised fine-tuning (SFT)
Adjust the response style to follow instructions
Tens to hundreds of thousands of examples
RLHF (reinforcement learning from human feedback)
Reinforcement learning toward the answers people prefer
Preference rankings assigned by people
DPO (direct preference optimization)
Reaches a goal similar to RLHF more simply, without a separate reward model
Preference ranking data (lighter to train than RLHF)
Lightweight fine-tuning (LoRA / QLoRA)
Trains only a small number of extra parameters instead of all the weights, specializing with few resources
Hundreds to thousands of examples
Model size and quantization
Model size and quantization is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
Performance generally rises with model size (parameter count), but so do memory and compute costs. Quantization is a technique that lowers the precision of the weights (for example 32-bit β 4-bit) to cut memory use, which makes it possible to run large models even on a single laptop.
Format/technique
Description
FP16 / BF16
The 16-bit floating point commonly used for training and inference (half the original precision)
INT8 / INT4 quantization
Approximates weights as 8-bit or 4-bit integers, cutting memory by a further half to three quarters
GGUF
The quantized model file format used by the llama.cpp family β widely used for local CPU/GPU inference
GPTQ / AWQ
The main algorithms for quantizing to the 4-bit level with minimal accuracy loss
Context Window and embeddings
Context Window and embeddings is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
The Context Window is the maximum number of tokens a model can refer to at once. To work with knowledge beyond that limit, you use RAG (Retrieval-Augmented Generation): embedding-based search (turning text into vectors) selects only the relevant content to put into the context. Instead of making the model memorize everything, RAG is like an open-book exam, where the reference material is found and handed over when needed.
Concept
Description
Context Window
The total tokens a model can refer to in one request (prompt + response)
Embedding
Text converted into a fixed-length vector that reflects its meaning
Cosine similarity
Computes semantic similarity from how closely two embedding vectors point in the same direction
Vector search
Finds the document embeddings closest to the question embedding to pick out relevant documents
RAG
Adds the documents found by vector search to the prompt to supply fresh or specialist knowledge
Decoding strategies: why the same model speaks differently
Here you look at Decoding strategies: why the same model speaks differently alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
At each step an LLM only outputs "a probability distribution over the next token"; which token is actually picked is decided by the decoding strategy. Even with the same model, these settings change the consistency and creativity of the answers considerably.
decoding_params.pyPYTHON
from openai import OpenAIclient = OpenAI()resp = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": "Write a short poem"}], temperature=0.9, # higher means more creative and varied wording top_p=0.95, # sample only within the top 95% of probability mass)print(resp.choices[0].message.content)
Strategy/parameter
Description
Effect
Greedy Decoding
Picks only the highest-probability token at every step
Consistent but repetitive and flat answers
Temperature
Controls how flat or peaked the probability distribution is
Low is deterministic, high is varied and random
Top-k
Samples only from the top k tokens by probability
Rules out strange tokens with very low probability
Top-p (nucleus)
Samples from the candidate set whose cumulative probability reaches p
The size of the candidate set adapts flexibly to the situation
Beam Search
Keeps several candidate paths at once and searches for the most plausible sentence overall
Good for tasks with a fairly clear right answer, such as translation
Prompt engineering basics
Here you look at Prompt engineering basics alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Even with the same model, result quality varies widely with how the prompt is put together. Below are the most basic patterns.
prompt_patterns.pyPYTHON
SYSTEM_PROMPT = "You are an assistant that answers only with accurate facts. If you do not know, say so."FEW_SHOT_EXAMPLES = """Question: What is 2 + 2?Answer: 4Question: Which planet is closest to Earth?Answer: Venus."""def build_prompt(question: str) -> str: return f"{FEW_SHOT_EXAMPLES}\nQuestion: {question}\nAnswer:"
Pattern
Description
Example
Zero-shot
Ask directly with no examples
"Classify the sentiment of this review: ..."
Few-shot
Provide a few input-output examples along with the question
2-3 examples + the new question
Chain-of-Thought (CoT)
Lead the model to work through its thinking step by step before answering
"Think step by step"
Role assignment (System Prompt)
Fix the role and constraints of the model at the start of the conversation
"You are a code reviewer..."
ReAct (reasoning + acting)
Alternates thinking (Reasoning) and tool calls (Acting)
The pattern where an Agent keeps reasoning while using a search tool
Hallucination and the limits of LLMs
Hallucination and the limits of LLMs is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
An LLM does not "look up" facts; it "generates" the most plausible next token based on the patterns it learned. That is why hallucination β confidently making up papers, APIs, or facts that do not exist as if they were real β can happen structurally.
Limit
Description
Mitigation
Hallucination
Generates content that sounds plausible but is not true
Supply grounding documents with RAG, require source citations, low temperature
No recent information
Knows nothing after the point its training data was collected (knowledge cutoff)
Add real-time information with a web search tool or RAG
Weak at numbers and arithmetic
Weak at exact calculation with large numbers and precise counting
Delegate to a calculator or code execution tool (Tool Use)
Inconsistency
The same question phrased differently can get a different answer
Lower the temperature, or sample several times and take a majority vote
The model landscape: what to choose when
The model landscape: what to choose when is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
The LLM ecosystem divides broadly into closed models, offered only through an API, and open-weight models, whose weights you can download and run yourself. Each has clear pros and cons, so choose according to the requirements of the project.
Category
Main models
Traits
Closed API
GPT (OpenAI), Claude (Anthropic), Gemini (Google)
Top-tier performance, no infrastructure to manage, data is sent outside
Open source/open weight
LLaMA (Meta), Mistral, Qwen, Gemma
Can be self-hosted, high freedom to fine-tune, you carry the infrastructure and operations burden
LLM evaluation and benchmarks
LLM evaluation and benchmarks is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
Saying "this model is better than that one" needs a yardstick. LLM evaluation divides broadly into standardized benchmark scores and custom evaluation matched to the real task.
Benchmark/method
What it measures
MMLU
Broad knowledge and reasoning through multiple-choice questions across 57 subjects
HumanEval
The ability to write working code from a function description
HellaSwag
Common-sense reasoning by choosing the sentence that follows naturally from the context
LLM-as-a-Judge
Having another strong LLM grade the quality of the responses
Human evaluation (Human Eval)
Real users and experts compare and rate the responses directly
Agents and Tool Use: when the LLM starts to act
Agents and Tool Use: when the LLM starts to act is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
A plain LLM only exchanges text, but combined with function/tool calling it becomes an Agent that decides for itself to perform external actions such as search, calculation, and DB lookups. The ReAct pattern seen earlier (repeating thought β action β observation) is the basic loop of that Agent behavior.
Concept
Description
Function/Tool Calling
The model reads a schema saying "call this tool in this situation" and decides by itself which tool to call with which arguments
ReAct loop
Approaches the goal by repeating thought (Reasoning) β tool call (Acting) β observing the result (Observation)
Multi-agent
Several Agents with different roles collaborate on or review one task
MCP (Model Context Protocol)
An open protocol that lets models connect to external tools and data sources in a standardized way
Safety and alignment
Safety and alignment is a point where the options diverge. Using the table to compare what each approach is for and how it differs in operation makes later decisions much easier.
The more powerful the model, the greater the potential for harmful answers, biased judgments, and misuse. Alignment refers to the whole process of matching the behavior of a model to human intent and values.
Concept
Description
RLHF
Aligns the model through reinforcement learning, using human preference rankings as the reward signal
Constitutional AI
Trains the model to critique and revise its own answers against a set of principles (a constitution), reducing human involvement
Red Teaming
Deliberately provoking weaknesses and harmful responses to find and fix them in advance
Guardrails
Application-level safeguards such as input/output filtering and blocking forbidden topics
LLM Fundamentals: frequently asked questions
What is an LLM?
An LLM (large language model) is trained on a huge amount of text and generates sentences by predicting the token that comes next. Answering questions, summarizing, and writing code are all the result of repeating that next-token prediction. That is why a model can produce text that sounds right but is wrong, and why information newer than its training data, or internal documents, has to be supplied separately.
What are tokens and the context window?
A token is the unit a model processes text in, often a piece smaller than a word. The context window is the maximum number of tokens the model can see in one request, and it includes the prompt, the conversation history, and the answer being generated. The same content can take a different number of tokens depending on the language, so measure cost and length with actual token counts.
Should I choose RAG or fine-tuning?
To make the model answer from internal documents or recent information it does not know, RAG is the fit. When documents change you only re-index, and the answer can show its sources. Fine-tuning is for consistently changing the format or tone of answers, or the way a specific task is carried out. Most teams start with better prompts and RAG, then add fine-tuning for what is still missing.
Why do hallucinations happen and how do I reduce them?
A model does not look facts up; it generates a plausible next token, so it writes fluent sentences even without evidence. To reduce this, put the supporting documents in the prompt (RAG), instruct the model to say it does not know when there is no evidence, and have it output sources so the answer can be checked. Hallucinations cannot be removed completely, so measure important outputs with an evaluation set and keep a human review step.