[Part 1] Cutting LLM API Costs: A 3-Layer Optimization Pattern with Input Compression and Model Selection

[Part 1] Stopping the LLM API Cost Bomb: A Master Guide to 3-Layer Cost Optimization Architecture Patterns

As service development with LLMs (Large Language Models) has become unprecedentedly active, the burden of operating costs has grown just as fast as expectations for performance. It often starts with “Wow, this feature is amazing!”—and before long you are facing a much more practical question: “Wait, this is costing more than 10 million won a month to run…?”

LLM API call costs are not mere consumable expenses. They are core operating expenses (OpEx) tied directly to the service’s sustainability. Beyond writing better prompts at the prompt-engineering stage, controlling cost at the system-architecture level has become an essential survival strategy.

This guide unpacks the cost structure of AI services and presents a practical architecture blueprint for cutting costs dramatically through a systematic 3-layer approach: input optimization $\rightarrow$ model selection strategy $\rightarrow$ system caching architecture.

💡 Introduction: “AI Services Hit by a Cost Bomb” — The Real Threat of LLM API Call Costs

LLM API pricing is fundamentally token-based. You are billed in proportion to the number of tokens in the input (Prompt) and the number of output (Completion) tokens the model generates.

The problem is that this cost is linear. Even a single user question can become expensive if the system internally runs complex reasoning or stuffs a massive document into context—costs can grow exponentially.

The goal is to keep top-tier performance while designing the most economical operating structure. That is not just about reducing API call volume; it requires architecture that blocks unnecessary tokens at the system level.

🧱 Layer 1: Input Optimization — Prompt Compression and Context Window Management

The first place to cut cost is the input. No matter how powerful the model, dumping irrelevant information in one shot is a shortcut to wasting money.

🔍 Beyond Simple Prompt Writing: The Art of Information Compression

Many developers paste an entire long document and ask, “Summarize this based on the content below.” That approach causes two problems. First, cost waste (sending unnecessary tokens). Second, information overload (the model misses the point or hits the context-window limit).

The key is to send the model only the necessary information, in the necessary form.

📝 Prompt Compression Example: Before & After

Practical tip: When using RAG (Retrieval-Augmented Generation), do not dump retrieved document chunks wholesale. Extract only summarized metadata or key keyword–description pairs from the retrieved chunks and use them as part of the prompt to maximize cost efficiency.

🧠 Layer 2: Strategic Model Selection — Small Model vs. Large Model: When to Use What?

The idea that “the smartest model will produce the best results” is one of the most dangerous cost traps. Performance (accuracy) and cost sit in a trade-off.

You should not chase the highest possible performance. You should choose the model that achieves the required performance at the lowest cost.

📊 Performance vs. Cost Model Selection Guidelines

💰 Calculating Cost by Tier

Price gaps between tiers differ by provider and generation and change often, so calculate them from the official price lists instead of memorizing ratios.

Key takeaway: Using a top-tier model for classification or simple summarization is like using a truck to deliver a few apples. A lightweight model can cut costs dramatically while still delivering sufficient performance.

🚀 3-Layer Summary and Next Steps

1. Layer 1 (Preprocessing): Compress input data as much as possible and strip unnecessary context to reduce token count. (The cheapest cost savings)
2. Layer 2 (Model selection): Choose the cheapest, most appropriate model for the difficulty of the task. (The largest cost savings)
3. Layer 3 (Caching): Reuse identical requests or store frequently used results to reduce the API calls themselves.

In the next installment, we will cover concrete implementation approaches that combine caching strategies (Layer 3) with RAG (Retrieval-Augmented Generation) patterns to reduce the number of API calls themselves.

Verifying Compression and Model Choice with Token Counts

"We shortened the prompt" isn't enough to know the cost effect. Measure before and after by token count and quality together to avoid over-compression that degrades answers.

Measurement

import tiktoken                     # OpenAI-family tokenizer
enc = tiktoken.get_encoding("o200k_base")
before = open("prompt_v1.txt").read()
after  = open("prompt_v2.txt").read()
print(len(enc.encode(before)), "→", len(enc.encode(after)))

Tokenizers differ by model, so the most accurate cost check is collecting the API response's usage field (input/output tokens) in logs. Korean text often uses more tokens than English for the same meaning, so measure directly if much of your traffic is Korean.

Why compression backfires

• Exception rules were removed from the system prompt and answer format became unstable
• Retrieved context was cut too far and unsupported answers increased
• The start of each request changed, losing the provider's prompt caching (reuse of identical prefixes)

Fix

• Put fixed instructions and examples at the very start and append per-user content after, to raise cache hits. Check caching conditions (minimum length, supported models) in the provider docs.
• Compare compressed and original versions on the same evaluation set and keep only reductions that don't lower quality.

Prevention

• Log input, output, and cached tokens separately and track cost per 1,000 requests per feature weekly.

Sources · checked 2026-10-04

• OpenAI models · OpenAI API pricing
• Anthropic Claude models overview · Anthropic pricing
• Google Gemini API models · Gemini API pricing

Example model names in this article come from the official model pages as of the check date. Model names, generations, and prices change every few months, so before adopting one, check the current models, prices, and deprecation schedules on these official pages.