Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment ยท evaluation ยท observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification ยท coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
๐Ÿง  AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
๐Ÿค– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
๐Ÿง  AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
๐ŸŒฑ Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
๐Ÿณ DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
๐Ÿงฑ Infrastructure
Infrastructure Intro & RoadmapNginxRedis
โ˜๏ธ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
๐ŸŽจ Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
๐Ÿ“ฑ Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
โš™๏ธ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
๐Ÿ’พ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
๐Ÿงช Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

ย 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
ยฉ 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
AIDevOps ยท AI Performance Guide

๐Ÿค– AI Service Performance Tuning A Practical Guide

Visitors

From reducing LLM API latency to tuning inference servers, speeding up RAG pipelines, and k6 load testing. Practical techniques for diagnosing and fixing the bottlenecks of production AI services.

โ† Blog
๐Ÿค–

Contents

0 / 11
  1. AI service performance metrics
  2. Reducing LLM latency
  3. Streaming responses
  4. Prompt caching
  5. Batch inference
  6. Inference server optimization
  7. Model quantization
  8. vLLM continuous batching
  9. RAG pipeline optimization
  10. k6 load testing for AI
  11. Monitoring & observability
Contents 11 sections
  1. AI service performance metrics
  2. Reducing LLM latency
  3. Streaming responses
  4. Prompt caching
  5. Batch inference
  6. Inference server optimization
  7. Model quantization
  8. vLLM continuous batching
  9. RAG pipeline optimization
  10. k6 load testing for AI
  11. Monitoring & observability

AI service performance metrics

AI service performance is measured with different metrics from an ordinary web API. Beyond plain response time, token generation speed and the delay to the first token decide the user experience.

MetricDescriptionTarget (production)
TTFT (Time To First Token)Time until the first token arrives< 500ms
TPS (Tokens Per Second)Tokens generated per second> 30 tok/s
E2E LatencyTotal time from request start to response completionp95 < 10s
ThroughputRequests handled per unit of timeDepends on the model/GPU
Queue DepthInference requests waiting in the queueAim to keep at 0
๐Ÿ’ก

Optimize TTFT first. In a streaming UI, users feel the response has arrived the moment the first token shows up. Even with a long E2E latency, a short TTFT greatly improves perceived performance.

Reducing LLM latency

LLM call latency comes from three stages: prompt processing (prefill), token generation (decode), and network transfer. Each stage can be optimized independently.

Streaming responses (SSE)

Instead of waiting for the full response, send tokens as soon as they are generated. Because TTFT is the heart of perceived performance, streaming is effectively mandatory.

app/api/chat/route.tsTYPESCRIPT
import { NextRequest } from 'next/server';

export async function POST(req: NextRequest) {
  const { messages } = await req.json();

  const upstream = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages,
      stream: true,          // enable streaming
      max_tokens: 1024,
    }),
  });

  // Pipe the upstream SSE straight through to the client
  return new Response(upstream.body, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
    },
  });
}
hooks/useStreamChat.tsTYPESCRIPT
export async function streamChat(messages: Message[], onChunk: (text: string) => void) {
  const res = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ messages }),
  });

  const reader = res.body!.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const lines = decoder.decode(value).split('\n');
    for (const line of lines) {
      if (!line.startsWith('data: ') || line === 'data: [DONE]') continue;
      const delta = JSON.parse(line.slice(6))
        ?.choices?.[0]?.delta?.content;
      if (delta) onChunk(delta);
    }
  }
}

Prompt caching (KV Cache)

When the same system prompt or context repeats, reusing the KV cache can sharply reduce prefill cost and TTFT. Both Anthropic and OpenAI support prompt caching.

PYTHON
import anthropic

client = anthropic.Anthropic()

# Apply cache_control to a long system prompt
# After the first call, TTFT drops by up to 85% on a cache hit
response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,           # thousands of tokens of context
            "cache_control": {"type": "ephemeral"} # mark for caching
        }
    ],
    messages=[{"role": "user", "content": user_query}],
)

# Check for a cache hit
usage = response.usage
print(f"Cache read tokens: {usage.cache_read_input_tokens}")
print(f"Cache creation tokens: {usage.cache_creation_input_tokens}")
โ„น๏ธ

Cache TTL: the Anthropic prompt cache lasts 5 minutes. It pays off when a RAG system keeps referring to the same documents, or when the accumulated context of a multi-turn conversation grows long.

Batch inference

Work that does not need a real-time response (classification, summarization, embedding) can be grouped into batches to raise GPU utilization and lower the unit cost.

PYTHON
import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def classify_batch(texts: list[str]) -> list[str]:
    """Classify several texts at once โ€” 3-5x the throughput of individual calls"""
    tasks = [
        client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Classify the text as positive/negative/neutral."},
                {"role": "user",   "content": text},
            ],
            max_tokens=10,
        )
        for text in texts
    ]
    responses = await asyncio.gather(*tasks)
    return [r.choices[0].message.content for r in responses]

# Run
results = asyncio.run(classify_batch(["Love it", "Not great", "It is okay"]))

Inference server optimization

When serving a self-hosted LLM (Llama, Mistral, Gemma, and so on), the choice of inference engine has a direct impact on performance.

EngineCharacteristicsBest for
vLLMPagedAttention, continuous batching, OpenAI API compatibleHigh-throughput production
OllamaGGUF support, simple install, mixed CPU/GPULocal development, small scale
TGI (HuggingFace)Flash Attention, tensor parallelism, many modelsThe HuggingFace ecosystem
llama.cppGGUF quantization, minimal resourcesEdge, low-spec environments

Model quantization

Quantizing an FP16 model to INT4/INT8 cuts VRAM use by up to 75% and speeds up inference. Quality loss is around 1-3% for INT8.

BASH
# AWQ quantization (INT4, minimal quality loss)
pip install autoawq

python -c "
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'meta-llama/Llama-3.1-8B-Instruct'
quant_path = 'llama-3.1-8b-awq-int4'

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

quant_config = {'zero_point': True, 'q_group_size': 128, 'w_bit': 4, 'version': 'GEMM'}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
"
MethodVRAM (8B model)SpeedQuality
FP1616 GBBaseline100%
INT8 (GPTQ)8 GB+20%~99%
INT4 (AWQ)4.5 GB+60%~97%
GGUF Q4_K_M4.8 GBRuns on CPU~96%

vLLM continuous batching settings

docker-compose.ymlYAML
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --quantization awq                  # AWQ INT4 quantization
      --max-model-len 8192               # context length
      --max-num-seqs 256                 # max concurrent sequences (continuous batching)
      --gpu-memory-utilization 0.90      # use 90% of GPU memory
      --enable-chunked-prefill           # process long prompts in chunks
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
๐Ÿ’ก

Raising --max-num-seqs increases throughput but lengthens the TTFT of individual requests. Set it low (32-64) for real-time chatbots and high (256-512) for batch processing.

RAG pipeline optimization

Most performance bottlenecks in RAG (Retrieval-Augmented Generation) are in the retrieval stage. Vector search is often slower than the LLM response itself.

PYTHON
import asyncio
from functools import lru_cache
from qdrant_client import AsyncQdrantClient
from openai import AsyncOpenAI

openai_client = AsyncOpenAI()
qdrant = AsyncQdrantClient(url="http://localhost:6333")

# Embedding cache โ€” avoids recomputing identical queries
@lru_cache(maxsize=1024)
def cache_key(query: str) -> str:
    return query.lower().strip()

embedding_cache: dict[str, list[float]] = {}

async def embed_with_cache(text: str) -> list[float]:
    key = cache_key(text)
    if key not in embedding_cache:
        res = await openai_client.embeddings.create(
            model="text-embedding-3-small",
            input=text,
        )
        embedding_cache[key] = res.data[0].embedding
    return embedding_cache[key]

async def retrieve(query: str, top_k: int = 5) -> list[dict]:
    # Parallelize embedding + search
    query_vec = await embed_with_cache(query)

    results = await qdrant.search(
        collection_name="docs",
        query_vector=query_vec,
        limit=top_k,
        with_payload=True,
        # HNSW parameter: higher ef is more accurate but slower (default 128)
        search_params={"hnsw_ef": 64},
    )
    return [r.payload for r in results]
OptimizationEffectTrade-off
Embedding cacheRepeated queries ~0msHigher memory use
Lower HNSW ef2-3x faster searchSlightly lower recall
Chunk size tuningBetter retrieval accuracyMore chunks
Hybrid searchKeyword + semantic combinedNeeds an added BM25 index

Load testing AI endpoints with k6

Unlike ordinary APIs, AI services have long response times and must handle streaming responses. Build realistic load scenarios in k6 that reflect these traits.

ai-load-test.jsJAVASCRIPT
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend, Counter } from 'k6/metrics';

// AI-specific custom metrics
const ttft    = new Trend('ai_time_to_first_token', true);
const totalTok = new Counter('ai_total_tokens');

export const options = {
  stages: [
    { duration: '1m', target: 10  }, // warm-up
    { duration: '3m', target: 50  }, // normal load
    { duration: '2m', target: 100 }, // peak load
    { duration: '1m', target: 0   }, // cool-down
  ],
  thresholds: {
    'http_req_duration':          ['p(95)<15000'], // within 15s for AI
    'ai_time_to_first_token':     ['p(95)<1000'],  // TTFT p95 < 1s
    'http_req_failed':            ['rate<0.01'],
  },
};

const PROMPTS = [
  "Explain three Redis caching strategies.",
  "What should I check when API responses are slow?",
  "Show me how to configure Kubernetes HPA.",
];

export default function () {
  const prompt = PROMPTS[Math.floor(Math.random() * PROMPTS.length)];
  const start  = Date.now();

  const res = http.post(
    'http://localhost:8000/v1/chat/completions',
    JSON.stringify({
      model: 'meta-llama/Llama-3.1-8B-Instruct',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 256,
      stream: false,
    }),
    { headers: { 'Content-Type': 'application/json' } },
  );

  check(res, {
    'status 200':         (r) => r.status === 200,
    'has content':        (r) => r.json()?.choices?.[0]?.message?.content?.length > 0,
  });

  const body = res.json();
  if (body?.usage) {
    totalTok.add(body.usage.completion_tokens);
    // non-streaming: TTFT โ‰ˆ total response time (conservative measurement)
    ttft.add(Date.now() - start);
  }

  sleep(Math.random() * 2 + 1); // 1-3s think time
}
โš ๏ธ

Streaming tests: k6 does not support SSE streams natively. To measure streaming TTFT accurately, either take the full response with stream: false or set up a separate WebSocket test.

BASH
# Run the test
k6 run ai-load-test.js

# Sample result
# ai_time_to_first_token  p(95)=842ms  โœ“
# http_req_duration        p(95)=6.2s   โœ“
# ai_total_tokens          12,840 tokens
# http_req_failed          0.00%        โœ“

Monitoring & observability

AI services need finer-grained monitoring than ordinary services. Token usage, GPU utilization, and queue wait time are the key signals to watch.

prometheus-ai-metrics.ymlYAML
# vLLM Prometheus metrics scrape config
scrape_configs:
  - job_name: 'vllm'
    static_configs:
      - targets: ['vllm:8000']
    metrics_path: '/metrics'

# Key PromQL alert rules
groups:
  - name: ai-service
    rules:
      - alert: HighTTFT
        expr: histogram_quantile(0.95, vllm:time_to_first_token_seconds_bucket) > 1
        for: 5m
        annotations:
          summary: "TTFT p95 exceeded 1 second"

      - alert: QueueBacklog
        expr: vllm:num_requests_waiting > 20
        for: 2m
        annotations:
          summary: "More than 20 requests are waiting in the inference queue"

      - alert: LowGPUUtilization
        expr: vllm:gpu_cache_usage_perc < 0.3
        for: 10m
        annotations:
          summary: "GPU cache utilization is below 30% โ€” consider scaling down"
MetricDescriptionThreshold (example)
vllm:time_to_first_token_secondsTTFT histogramp95 < 1s
vllm:num_requests_runningRequests currently being processedAt or below max-num-seqs
vllm:num_requests_waitingRequests waiting in the queue< 20
vllm:gpu_cache_usage_percGPU usage of the KV cache> 0.7
vllm:generation_tokens_totalTotal generated tokensFor cost tracking
๐Ÿ’ก

Optimizing performance per cost: if GPU cache usage stays above 90%, shorten the context length or add GPUs. Below 30% you are over-provisioned, so consider scaling down.