From COBOL to GPT: A Three-Stage Architecture Pattern Guide for Connecting Legacy Systems and LLMs

From COBOL to GPT: A Three-Stage Architecture Pattern Guide for Connecting Legacy Systems and LLMs

"Our company's core business logic lives on a COBOL mainframe. We want to introduce a modern LLM to automate customer inquiries—but how do we connect these two worlds?"

If you have asked that question, you are already standing at the intersection of the enterprise's most valuable asset (core business logic) and its hottest technology trend (generative AI).

Reality, however, is unforgiving. Between the mainframe's tangled transaction code, decades of accumulated XML-based data formats, and the clean JSON API calls that modern LLMs expect, there is a massive wall of time. Closing that gap is the central challenge of enterprise AI architecture.

This article is not a recitation of abstract concepts. To solve the legacy-to-LLM integration problem that so many IT architects face, it presents a practically applicable three-stage architecture pattern (blueprint). Once you understand this roadmap, you can accelerate AI innovation without fearing technical debt.

🧱 Stage 1: Data Extraction and Standardization (The Extraction Layer)

The first problem to solve is the shape of the data. LLMs understand text best, but legacy system data often exists only as byte streams or fixed field orders.

The Legacy Data Dilemma: Structured, Unstructured, and Semi-Structured

Legacy data typically appears in these forms.

• Structured: Database tables (but access methods are often complex).
• Semi-structured: XML, JSON (but schemas are highly complex or inconsistent).
• Unstructured: Historical transaction log files, scanned documents (OCR required).

The goal of Stage 1 is to transform this data into a standardized schema that an LLM can understand.

💡 Solution Pattern: API Gateway and Transformation Layer

The core principle of this stage is no direct access. Wiring an LLM straight into legacy core logic seriously threatens system stability. Instead, we must put a safe gateway in front of it.

1. API Gateway role: Accepts external requests and converts them into a form the legacy system understands (for example, a specific legacy API calling convention).
2. Transformation Layer: This layer is the intelligent translator. For instance, it takes a CUST_ID field coming from COBOL and a <CustomerID> field coming from XML and maps both to a unified internal JSON schema field named customer_uuid.

graph LR
    A[외부 요청 (LLM/UI)] --> B(API Gateway);
    B --> C{Transformation Layer};
    C --> D[레거시 시스템 (COBOL/DB)];
    D --> C;
    C --> E[표준화된 JSON 응답];

Core technology stack recommendation: Use an API Gateway (Kong, Apigee) and a message queue (Kafka, RabbitMQ) to guarantee asynchronous data processing.

⚙️ Stage 2: Building an Intelligent Intermediary (The Orchestration Layer)

Once data is extracted and standardized, you must control the flow of requests. LLMs are excellent at reasoning about what to do, but they have no authority to actually execute anything.

The orchestration layer fills that gap. This layer acts as the conductor between the LLM and the legacy system.

🔍 Applying a Legacy-Friendly Version of Function Calling

Modern LLMs can invoke external APIs through Function Calling. We need to reinterpret that idea for a legacy environment.

When the orchestration layer receives the LLM's output, it concludes, "The user asked question A, so I should call the get_customer_status(customer_uuid) function that we standardized in Stage 1," and then it runs that function.

⚠️ Warning: Direct Connection (Bad) vs. Via Orchestration Layer (Good)

Core technology stack recommendation: Use LLM frameworks (LangChain, LlamaIndex) and a workflow engine (Temporal, Apache Airflow) to define the sequence and exception handling of complex business logic in code.

🧠 Stage 3: Maximizing LLM Value (The Intelligence Layer)

With data standardized and system call flows under control, it is time for the LLM to show its true value. It should go beyond simply answering "What is the customer number?" and address why and so what should we do next.

The heart of this stage is applying the RAG (Retrieval-Augmented Generation) pattern to the legacy knowledge base.

📚 Scenario: Generating Answers from Legacy Manuals

Suppose a customer asks, "What business logic does this abnormal transaction log from last month (XML attached) represent?"

1. Retrieval: The system analyzes the attached log and searches hundreds of pages of internally stored operations manuals (PDF, Wiki) for related keywords and patterns.
2. Augmentation: It pulls the retrieved manual clauses (Context) and attaches them to the prompt together with the user's question.
3. Generation: Based on this [question + manual clauses], the LLM produces an evidence-based answer such as, "This log should be processed according to section 3.2.1 of the manual, and an appeal to the responsible department is required."

This process is the core mechanism that prevents LLM hallucination and delivers grounded answers.

🚀 Summary: Three-Stage Data Flow

Organically connecting these three stages is precisely the role of the AI agent, and that is the key to successfully integrating AI with legacy systems.