vLLM vs TGI: Comparing LLM Inference Engines by Batch Size and Concurrent Requests

vLLM vs TGI In-Depth Comparison: A Complete Guide to LLM Inference Optimization by Batch Size and Concurrent Requests

When you build a service on top of an LLM, the hardest—and most expensive—part is inference. Training a model and answering live user requests require completely different optimization points. Loading the model is not enough. You have to control two variables—maximum concurrent users (Concurrency) and optimal batch size (Batch Size)—and design an architecture that fully utilizes GPU resources.

This guide is for practicing ML engineers and MLOps developers. It shows how to systematically benchmark inference performance with industry-standard vLLM and TGI (Text Generation Inference), and how to find memory bottlenecks in practice.

Why Is LLM Inference Performance Optimization So Hard?

Understanding why LLM inference slows down is the first step toward optimizing it. In typical compute workloads, increasing input size (batch size) tends to raise throughput linearly. LLMs add extra complexity because of token-by-token generation.

💡 Core Principle: The Magic of KV Cache and PagedAttention

An LLM generates text by repeatedly predicting one token at a time. Along the way it must store the key (Key) and value (Value) vectors of every previously generated token—this is the KV Cache.

If you manage the KV Cache poorly, GPU memory is exhausted almost immediately. The innovations vLLM and TGI brought here are PagedAttention and Continuous Batching.

1. PagedAttention (memory-management innovation): Traditional allocation reserved memory in fixed-size blocks, wasting far more memory than was actually used. PagedAttention borrows the OS virtual-memory idea and manages the KV Cache in page-sized units. It allocates and frees only what is needed, which sharply reduces memory fragmentation and maximizes memory efficiency.
2. Continuous Batching (throughput maximization): Traditional batching waited for the entire batch until the slowest request finished. If 1 of 10 requests was very slow, the other 9 had to wait at that pace. Continuous Batching breaks that model: as soon as a request completes, its slot is freed and a new request is admitted. That minimizes GPU idle time and maximizes throughput.

Thanks to these two techniques, the goal is no longer just “make the batch bigger.” It is efficient resource allocation and zero idle time.

Building a Practical Benchmark Environment: Using vLLM vs TGI

To compare performance you need more than API calls—you need a system that controls variables and measures repeatedly. You must vary two core parameters:

1. Batch Size (BS): How many requests are processed at once (the size of the set processed at the same moment).
2. Concurrency (C): The maximum number of requests the system can hold and process at the same time (concurrent-request capacity).

🛠️ Runnable Benchmark Logic (Python Snippet)

The snippet below uses vLLM as an example; the same core logic applies to TGI. It tests combinations in a for loop and provides a framework for measuring system resources.

import time
import psutil
import os
from vllm import LLM, SamplingParams
# from nvidia.system import nvidia_smi #

# --- settings ---
MODEL_NAME = "meta-llama/Llama-2-7b-hf" # path
MAX_BATCH_SIZE = 16
MAX_CONCURRENCY = 32
START_TOKEN_COUNT = 100 # Request

# vLLM (GPU check )
print(" ... ( GPU )")
llm = LLM(model=MODEL_NAME, dtype="half", trust_remote_code=True)
sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=START_TOKEN_COUNT)

# Result save
results = []

print("\n=====================================================")
print("🚀 when: BS Concurrency measure")
print("=====================================================")

# 1. (BS) when Request (C)
for bs in range(1, MAX_BATCH_SIZE + 1, 4): # BS 4 when
 for conc in range(1, MAX_CONCURRENCY + 1, 8): # Concurrency 8 when

 print(f"\n[ ] Batch Size: {bs}, Concurrency: {conc}")

 # --- run ---
        start_time = time.time()

 # when: 'conc' Request 'bs' when
 # (asyncio) 'conc' Request .
        dummy_inputs = ["A simple prompt for testing performance."] * conc

 # vLLM generate_batch .
        outputs = llm.generate_batch(dummy_inputs, sampling_params)

        end_time = time.time()

 # --- measure ---
        elapsed_time = end_time - start_time
        total_tokens_generated = len(outputs) * START_TOKEN_COUNT

 # (Throughput): /
        throughput = total_tokens_generated / elapsed_time

 # measure ( psutil.Process.memory_info )
 # , 'Concurrency' 'Model Size' .
 estimated_memory_usage = (conc * 0.005) + (bs * 0.001) #

        results.append({
            "BS": bs,
            "Concurrency": conc,
            "Throughput (Tokens/sec)": throughput,
            "Memory Usage (GB)": estimated_memory_usage
        })

print("\n=====================================================")
print("✅ . Result .")

📊 Performance Comparison Table: Finding the Optimal Combination

Structuring the data from the iterative tests above yields a table like the following. Values vary a lot by model, hardware, and code optimizations, so you must fill them in from your own tests.

💡 Analysis points:

1. Throughput: The combination with the highest value is likely the current system’s sweet spot.
2. Memory Overhead: A sharp jump in memory use is an OOM warning. Stay below that point.

🚀 Conclusion and Optimization Guidelines

Optimization means finding the point that maximizes throughput while keeping memory overhead stable.

1. Batch Size vs. Concurrency:

   • Increase batch size: Process more data at once to fully exploit GPU parallelism. (Maximize GPU utilization)
   • Increase concurrency: Serve many independent requests at once to raise overall system throughput. (Maximize system throughput)
   • Tune both: Combine the two so the GPU is not overloaded and requests still complete without delay.

2. Use a serving framework:

   • Use vLLM or TGI: If you are serving LLM inference, switching from Hugging Face’s default libraries to an optimized serving stack such as vLLM or Text Generation Inference (TGI) has the largest impact. They use advanced memory management such as PagedAttention.

The most important next step is to simulate your production traffic pattern (concurrent users, average request size) and run these tests iteratively.