From reducing LLM API latency to tuning inference servers, speeding up RAG pipelines, and k6 load testing. Practical techniques for diagnosing and fixing the bottlenecks of production AI services.
AI service performance is measured with different metrics from an ordinary web API. Beyond plain response time, token generation speed and the delay to the first token decide the user experience.
| Metric | Description | Target (production) |
|---|---|---|
| TTFT (Time To First Token) | Time until the first token arrives | < 500ms |
| TPS (Tokens Per Second) | Tokens generated per second | > 30 tok/s |
| E2E Latency | Total time from request start to response completion | p95 < 10s |
| Throughput | Requests handled per unit of time | Depends on the model/GPU |
| Queue Depth | Inference requests waiting in the queue | Aim to keep at 0 |
Optimize TTFT first. In a streaming UI, users feel the response has arrived the moment the first token shows up. Even with a long E2E latency, a short TTFT greatly improves perceived performance.
LLM call latency comes from three stages: prompt processing (prefill), token generation (decode), and network transfer. Each stage can be optimized independently.
Instead of waiting for the full response, send tokens as soon as they are generated. Because TTFT is the heart of perceived performance, streaming is effectively mandatory.
import { NextRequest } from 'next/server';
export async function POST(req: NextRequest) {
const { messages } = await req.json();
const upstream = await fetch('https://api.openai.com/v1/chat/completions', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
},
body: JSON.stringify({
model: 'gpt-4o-mini',
messages,
stream: true, // enable streaming
max_tokens: 1024,
}),
});
// Pipe the upstream SSE straight through to the client
return new Response(upstream.body, {
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
},
});
}export async function streamChat(messages: Message[], onChunk: (text: string) => void) {
const res = await fetch('/api/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ messages }),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const lines = decoder.decode(value).split('\n');
for (const line of lines) {
if (!line.startsWith('data: ') || line === 'data: [DONE]') continue;
const delta = JSON.parse(line.slice(6))
?.choices?.[0]?.delta?.content;
if (delta) onChunk(delta);
}
}
}When the same system prompt or context repeats, reusing the KV cache can sharply reduce prefill cost and TTFT. Both Anthropic and OpenAI support prompt caching.
import anthropic
client = anthropic.Anthropic()
# Apply cache_control to a long system prompt
# After the first call, TTFT drops by up to 85% on a cache hit
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
system=[
{
"type": "text",
"text": LONG_SYSTEM_PROMPT, # thousands of tokens of context
"cache_control": {"type": "ephemeral"} # mark for caching
}
],
messages=[{"role": "user", "content": user_query}],
)
# Check for a cache hit
usage = response.usage
print(f"Cache read tokens: {usage.cache_read_input_tokens}")
print(f"Cache creation tokens: {usage.cache_creation_input_tokens}")Cache TTL: the Anthropic prompt cache lasts 5 minutes. It pays off when a RAG system keeps referring to the same documents, or when the accumulated context of a multi-turn conversation grows long.
Work that does not need a real-time response (classification, summarization, embedding) can be grouped into batches to raise GPU utilization and lower the unit cost.
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI()
async def classify_batch(texts: list[str]) -> list[str]:
"""Classify several texts at once โ 3-5x the throughput of individual calls"""
tasks = [
client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Classify the text as positive/negative/neutral."},
{"role": "user", "content": text},
],
max_tokens=10,
)
for text in texts
]
responses = await asyncio.gather(*tasks)
return [r.choices[0].message.content for r in responses]
# Run
results = asyncio.run(classify_batch(["Love it", "Not great", "It is okay"]))When serving a self-hosted LLM (Llama, Mistral, Gemma, and so on), the choice of inference engine has a direct impact on performance.
| Engine | Characteristics | Best for |
|---|---|---|
| vLLM | PagedAttention, continuous batching, OpenAI API compatible | High-throughput production |
| Ollama | GGUF support, simple install, mixed CPU/GPU | Local development, small scale |
| TGI (HuggingFace) | Flash Attention, tensor parallelism, many models | The HuggingFace ecosystem |
| llama.cpp | GGUF quantization, minimal resources | Edge, low-spec environments |
Quantizing an FP16 model to INT4/INT8 cuts VRAM use by up to 75% and speeds up inference. Quality loss is around 1-3% for INT8.
# AWQ quantization (INT4, minimal quality loss)
pip install autoawq
python -c "
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = 'meta-llama/Llama-3.1-8B-Instruct'
quant_path = 'llama-3.1-8b-awq-int4'
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
quant_config = {'zero_point': True, 'q_group_size': 128, 'w_bit': 4, 'version': 'GEMM'}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
"| Method | VRAM (8B model) | Speed | Quality |
|---|---|---|---|
| FP16 | 16 GB | Baseline | 100% |
| INT8 (GPTQ) | 8 GB | +20% | ~99% |
| INT4 (AWQ) | 4.5 GB | +60% | ~97% |
| GGUF Q4_K_M | 4.8 GB | Runs on CPU | ~96% |
services:
vllm:
image: vllm/vllm-openai:latest
runtime: nvidia
environment:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
command: >
--model meta-llama/Llama-3.1-8B-Instruct
--quantization awq # AWQ INT4 quantization
--max-model-len 8192 # context length
--max-num-seqs 256 # max concurrent sequences (continuous batching)
--gpu-memory-utilization 0.90 # use 90% of GPU memory
--enable-chunked-prefill # process long prompts in chunks
ports:
- "8000:8000"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]Raising --max-num-seqs increases throughput but lengthens the TTFT of individual requests. Set it low (32-64) for real-time chatbots and high (256-512) for batch processing.
Most performance bottlenecks in RAG (Retrieval-Augmented Generation) are in the retrieval stage. Vector search is often slower than the LLM response itself.
import asyncio
from functools import lru_cache
from qdrant_client import AsyncQdrantClient
from openai import AsyncOpenAI
openai_client = AsyncOpenAI()
qdrant = AsyncQdrantClient(url="http://localhost:6333")
# Embedding cache โ avoids recomputing identical queries
@lru_cache(maxsize=1024)
def cache_key(query: str) -> str:
return query.lower().strip()
embedding_cache: dict[str, list[float]] = {}
async def embed_with_cache(text: str) -> list[float]:
key = cache_key(text)
if key not in embedding_cache:
res = await openai_client.embeddings.create(
model="text-embedding-3-small",
input=text,
)
embedding_cache[key] = res.data[0].embedding
return embedding_cache[key]
async def retrieve(query: str, top_k: int = 5) -> list[dict]:
# Parallelize embedding + search
query_vec = await embed_with_cache(query)
results = await qdrant.search(
collection_name="docs",
query_vector=query_vec,
limit=top_k,
with_payload=True,
# HNSW parameter: higher ef is more accurate but slower (default 128)
search_params={"hnsw_ef": 64},
)
return [r.payload for r in results]| Optimization | Effect | Trade-off |
|---|---|---|
| Embedding cache | Repeated queries ~0ms | Higher memory use |
| Lower HNSW ef | 2-3x faster search | Slightly lower recall |
| Chunk size tuning | Better retrieval accuracy | More chunks |
| Hybrid search | Keyword + semantic combined | Needs an added BM25 index |
Unlike ordinary APIs, AI services have long response times and must handle streaming responses. Build realistic load scenarios in k6 that reflect these traits.
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend, Counter } from 'k6/metrics';
// AI-specific custom metrics
const ttft = new Trend('ai_time_to_first_token', true);
const totalTok = new Counter('ai_total_tokens');
export const options = {
stages: [
{ duration: '1m', target: 10 }, // warm-up
{ duration: '3m', target: 50 }, // normal load
{ duration: '2m', target: 100 }, // peak load
{ duration: '1m', target: 0 }, // cool-down
],
thresholds: {
'http_req_duration': ['p(95)<15000'], // within 15s for AI
'ai_time_to_first_token': ['p(95)<1000'], // TTFT p95 < 1s
'http_req_failed': ['rate<0.01'],
},
};
const PROMPTS = [
"Explain three Redis caching strategies.",
"What should I check when API responses are slow?",
"Show me how to configure Kubernetes HPA.",
];
export default function () {
const prompt = PROMPTS[Math.floor(Math.random() * PROMPTS.length)];
const start = Date.now();
const res = http.post(
'http://localhost:8000/v1/chat/completions',
JSON.stringify({
model: 'meta-llama/Llama-3.1-8B-Instruct',
messages: [{ role: 'user', content: prompt }],
max_tokens: 256,
stream: false,
}),
{ headers: { 'Content-Type': 'application/json' } },
);
check(res, {
'status 200': (r) => r.status === 200,
'has content': (r) => r.json()?.choices?.[0]?.message?.content?.length > 0,
});
const body = res.json();
if (body?.usage) {
totalTok.add(body.usage.completion_tokens);
// non-streaming: TTFT โ total response time (conservative measurement)
ttft.add(Date.now() - start);
}
sleep(Math.random() * 2 + 1); // 1-3s think time
}Streaming tests: k6 does not support SSE streams natively. To measure streaming TTFT accurately, either take the full response with stream: false or set up a separate WebSocket test.
# Run the test
k6 run ai-load-test.js
# Sample result
# ai_time_to_first_token p(95)=842ms โ
# http_req_duration p(95)=6.2s โ
# ai_total_tokens 12,840 tokens
# http_req_failed 0.00% โAI services need finer-grained monitoring than ordinary services. Token usage, GPU utilization, and queue wait time are the key signals to watch.
# vLLM Prometheus metrics scrape config
scrape_configs:
- job_name: 'vllm'
static_configs:
- targets: ['vllm:8000']
metrics_path: '/metrics'
# Key PromQL alert rules
groups:
- name: ai-service
rules:
- alert: HighTTFT
expr: histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) > 1
for: 5m
annotations:
summary: "TTFT p95 exceeded 1 second"
- alert: QueueBacklog
expr: vllm:num_requests_waiting > 20
for: 2m
annotations:
summary: "More than 20 requests are waiting in the inference queue"
- alert: LowGPUUtilization
expr: vllm:gpu_cache_usage_perc < 0.3
for: 10m
annotations:
summary: "GPU cache utilization is below 30% โ consider scaling down"| Metric | Description | Threshold (example) |
|---|---|---|
| vllm:time_to_first_token_seconds | TTFT histogram | p95 < 1s |
| vllm:num_requests_running | Requests currently being processed | At or below max-num-seqs |
| vllm:num_requests_waiting | Requests waiting in the queue | < 20 |
| vllm:gpu_cache_usage_perc | GPU usage of the KV cache | > 0.7 |
| vllm:generation_tokens_total | Total generated tokens | For cost tracking |
Optimizing performance per cost: if GPU cache usage stays above 90%, shorten the context length or add GPUs. Below 30% you are over-provisioned, so consider scaling down.