vLLM vs TGI In-Depth Comparison: A Complete Guide to LLM Inference Optimization by Batch Size and Concurrent Requests

vLLM vs TGI In-Depth Comparison: A Complete Guide to LLM Inference Optimization by Batch Size and Concurrent Requests

When you build a service on top of an LLM, the hardest—and most expensive—part is inference. Training a model and answering live user requests require completely different optimization points. Loading the model is not enough. You have to control two variables—maximum concurrent users (Concurrency) and optimal batch size (Batch Size)—and design an architecture that fully utilizes GPU resources.

This guide is for practicing ML engineers and MLOps developers. It shows how to systematically benchmark inference performance with industry-standard vLLM and TGI (Text Generation Inference), and how to find memory bottlenecks in practice.

Why Is LLM Inference Performance Optimization So Hard?

Understanding why LLM inference slows down is the first step toward optimizing it. In typical compute workloads, increasing input size (batch size) tends to raise throughput linearly. LLMs add extra complexity because of token-by-token generation.

💡 Core Principle: The Magic of KV Cache and PagedAttention

An LLM generates text by repeatedly predicting one token at a time. Along the way it must store the key (Key) and value (Value) vectors of every previously generated token—this is the KV Cache.

If you manage the KV Cache poorly, GPU memory is exhausted almost immediately. The innovations vLLM and TGI brought here are PagedAttention and Continuous Batching.

1. PagedAttention (memory-management innovation): Traditional allocation reserved memory in fixed-size blocks, wasting far more memory than was actually used. PagedAttention borrows the OS virtual-memory idea and manages the KV Cache in page-sized units. It allocates and frees only what is needed, which sharply reduces memory fragmentation and maximizes memory efficiency.
2. Continuous Batching (throughput maximization): Traditional batching waited for the entire batch until the slowest request finished. If 1 of 10 requests was very slow, the other 9 had to wait at that pace. Continuous Batching breaks that model: as soon as a request completes, its slot is freed and a new request is admitted. That minimizes GPU idle time and maximizes throughput.

Thanks to these two techniques, the goal is no longer just “make the batch bigger.” It is efficient resource allocation and zero idle time.

Building a Practical Benchmark Environment: Using vLLM vs TGI

To compare performance you need more than API calls—you need a system that controls variables and measures repeatedly. You must vary two core parameters:

1. Batch Size (BS): How many requests are processed at once (the size of the set processed at the same moment).
2. Concurrency (C): The maximum number of requests the system can hold and process at the same time (concurrent-request capacity).

🛠️ Runnable Benchmark Logic (Python Snippet)

The snippet below uses vLLM as an example; the same core logic applies to TGI. It tests combinations in a for loop and provides a framework for measuring system resources.

import time
import psutil
import os
from vllm import LLM, SamplingParams
# from nvidia.system import nvidia_smi # 실제 환경에 맞게 사용

# --- 설정 변수 ---
MODEL_NAME = "meta-llama/Llama-2-7b-hf" # 테스트할 모델 경로
MAX_BATCH_SIZE = 16
MAX_CONCURRENCY = 32
START_TOKEN_COUNT = 100 # 각 요청이 생성할 평균 토큰 수

# vLLM 모델 로드 (GPU 메모리 점유 확인 필요)
print("모델 로딩 중... (이 과정에서 GPU 메모리 할당이 발생합니다)")
llm = LLM(model=MODEL_NAME, dtype="half", trust_remote_code=True)
sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=START_TOKEN_COUNT)

# 벤치마크 결과 저장용 리스트
results = []

print("\n=====================================================")
print("🚀 벤치마크 시작: BS와 Concurrency 변화에 따른 측정")
print("=====================================================")

# 1. 배치 사이즈(BS)와 동시 요청 수(C)를 조합하여 반복 테스트
for bs in range(1, MAX_BATCH_SIZE + 1, 4): # BS는 4 단위로 증가시켜 테스트
    for conc in range(1, MAX_CONCURRENCY + 1, 8): # Concurrency는 8 단위로 증가시켜 테스트
        
        print(f"\n[테스트 조합] Batch Size: {bs}, Concurrency: {conc}")
        
        # --- 벤치마크 실행 로직 ---
        start_time = time.time()
        
        # 시뮬레이션: 'conc'개의 요청을 'bs' 크기의 배치로 처리하는 과정을 시뮬레이션
        # 실제로는 비동기(asyncio)를 사용하여 'conc'개의 독립적인 요청을 유지하며 부하를 줍니다.
        dummy_inputs = ["A simple prompt for testing performance."] * conc
        
        # vLLM의 generate_batch를 사용하여 부하를 줍니다.
        outputs = llm.generate_batch(dummy_inputs, sampling_params)
        
        end_time = time.time()
        
        # --- 성능 측정 및 기록 ---
        elapsed_time = end_time - start_time
        total_tokens_generated = len(outputs) * START_TOKEN_COUNT
        
        # 처리량 (Throughput): 토큰/초
        throughput = total_tokens_generated / elapsed_time
        
        # 메모리 사용량 측정 (실제로는 주기적으로 psutil.Process().memory_info()를 체크해야 함)
        # 여기서는 단순화하여, 메모리 사용량은 'Concurrency'와 'Model Size'에 비례한다고 가정합니다.
        estimated_memory_usage = (conc * 0.005) + (bs * 0.001) # 가상 계산
        
        results.append({
            "BS": bs, 
            "Concurrency": conc, 
            "Throughput (Tokens/sec)": throughput, 
            "Memory Usage (GB)": estimated_memory_usage
        })

print("\n=====================================================")
print("✅ 벤치마크 완료. 결과 분석 단계로 이동합니다.")

📊 Performance Comparison Table: Finding the Optimal Combination

Structuring the data from the iterative tests above yields a table like the following. Values vary a lot by model, hardware, and code optimizations, so you must fill them in from your own tests.

💡 Analysis points:

1. Throughput: The combination with the highest value is likely the current system’s sweet spot.
2. Memory Overhead: A sharp jump in memory use is an OOM warning. Stay below that point.

🚀 Conclusion and Optimization Guidelines

Optimization means finding the point that maximizes throughput while keeping memory overhead stable.

1. Batch Size vs. Concurrency:

   • Increase batch size: Process more data at once to fully exploit GPU parallelism. (Maximize GPU utilization)
   • Increase concurrency: Serve many independent requests at once to raise overall system throughput. (Maximize system throughput)
   • Tune both: Combine the two so the GPU is not overloaded and requests still complete without delay.

2. Use a serving framework:

   • Use vLLM or TGI: If you are serving LLM inference, switching from Hugging Face’s default libraries to an optimized serving stack such as vLLM or Text Generation Inference (TGI) has the largest impact. They use advanced memory management such as PagedAttention.

The most important next step is to simulate your production traffic pattern (concurrent users, average request size) and run these tests iteratively.