[Complete Code Example Guide] Building Your Own RAG System with LangChain & LlamaIndex

[Complete Code Example Guide] Building Your Own RAG System with LangChain & LlamaIndex

"LLMs are all-powerful? Not quite. But to use an LLM well, you need a knowledge base."

The pace of LLM (large language model) progress over the past few years has been remarkable. They can seem almost magical at answering complex questions, writing, and coding. But as developers, when we try to put this powerful tool into a real product, we often hit a wall.

"What if the model's knowledge isn't up to date?" "What if it doesn't know a specific policy in our internal company manuals?" "Won't it sometimes make things up (hallucination)?"

These problems are exactly why RAG (Retrieval-Augmented Generation) is needed. RAG is the process of having an LLM answer by consulting an external, trustworthy knowledge base.

Theory alone is not enough. Today I will turn the abstract idea of RAG into working code blocks and a step-by-step workflow—a complete blueprint you can follow today and actually produce a result.

📚 Step 1: Data Preparation and Embedding (Building the Knowledge Base)

The success of a RAG system depends 80% on how well you build this knowledge base. Think of it as classifying books in a library and organizing them so they are easy to search.

We start with unstructured documents such as PDFs and DOCX files. The first goal is to convert those documents into numeric vectors the LLM can work with.

🧱 Workflow Overview: Load $\rightarrow$ Split $\rightarrow$ Embed $\rightarrow$ Store

1. Document Loader: Load files of various formats (PDF, TXT, etc.).
2. Text Splitter: Split the loaded documents into appropriately sized chunks that fit the LLM's context window. Too large and you get noise; too small and you lose context.
3. Embedding: Convert each chunk into a high-dimensional vector (an array of numbers) using an embedding model (e.g., OpenAI text-embedding-ada-002).
4. Vector Store: Store these vectors in a specialized database (Vector DB) that is easy to search.

💻 Hands-on Code: Loading Data and Vectorizing (LlamaIndex Example)

In a real environment, you need to install libraries first, e.g. pip install llama-index pypdf chromadb openai.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
import os

# 1. 가상 데이터 준비 (실제로는 PDF 폴더를 지정합니다)
# os.makedirs("data", exist_ok=True)
# print("💡 'data' 폴더에 테스트용 매뉴얼 PDF 파일을 넣어주세요.")

# 2. Document Loader: 로컬 폴더에서 문서 로드
print("✅ 1. 문서 로딩 시작...")
try:
    documents = SimpleDirectoryReader("data").load_data()
    print(f"✨ 총 {len(documents)}개의 문서를 성공적으로 로드했습니다.")
except FileNotFoundError:
    print("🚨 'data' 폴더를 찾을 수 없습니다. 테스트 파일을 넣어주세요.")
    exit()

# 3. Vector Store 및 Index 생성
# ChromaDB를 사용하며, 로컬에 벡터 DB를 구축합니다.
print("✅ 2. ChromaDB 연결 및 Index 구축 시작...")
db = chromadb.EphemeralClient() # 임시 클라이언트 사용
chroma_collection = db.get_or_create_collection("company_manual_index")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

# 4. Index 생성 (자동으로 청킹, 임베딩, 저장까지 처리)
# LlamaIndex는 이 과정을 매우 간결하게 처리해줍니다.
index = VectorStoreIndex.from_documents(
    documents,
    vector_store=vector_store,
    show_progress=True
)

print("\n🎉 데이터베이스 구축 완료! 이제 검색 준비가 끝났습니다.")

💡 Developer debugging tip: If you hit an Embedding Model related error, first check that your API key (OPENAI_API_KEY, etc.) is set correctly in the environment variables, and that the model is actually available (e.g., whether you have a paid API).

🚀 Step 2: Building Orchestration (Completing the Q&A Pipeline)

Once the knowledge base is ready, you need to build the "brain" that takes a user question and generates an answer. That process is orchestration.

Core flow: $$\text{User Query} \xrightarrow{\text{Embedding}} \text{Vector Search} \xrightarrow{\text{Retrieval}} \text{Context Documents} \xrightarrow{\text{Prompt Augmentation}} \text{LLM} \xrightarrow{\text{Answer}} \text{Final Answer}$$

📊 LangChain vs. LlamaIndex: Which Tool Is Right for You?

Both frameworks are well suited to RAG, but they differ in philosophy.

Bottom line: Both are excellent for laying the foundation of a RAG system, but if you are focused purely on document-based knowledge retrieval, LlamaIndex can be more intuitive.

💻 Code Example (LangChain-based)

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA

# 1. 임베딩 및 벡터 저장소 로드 (이전 단계에서 생성된 벡터 DB 사용 가정)
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(
    documents=loaded_documents, # 미리 로드된 문서 객체
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# 2. 검색기(Retriever) 설정
retriever = vectorstore.as_retriever(search_kwargs={"k": 3}) # 상위 3개 문서 검색

# 3. QA 체인 생성 (검색된 문서를 기반으로 답변 생성)
llm = ChatOpenAI(model_name="gpt-4o")
qa_chain = RetrievalQA.from_chain_type(
    llm=llm, 
    retriever=retriever, 
    return_source_documents=True # 출처 문서도 함께 반환받기
)

# 4. 질문 실행
query = "최근 시장 동향에 대한 핵심 요약은 무엇인가요?"
result = qa_chain({"query": query})

print(f"🤖 답변: {result['result']}")
print("\n📚 출처 문서:")
for doc in result['source_documents']:
    print(f"- {doc.metadata.get('source')}: {doc.page_content[:50]}...")

💡 Advanced Topics: Raising the Quality of Your RAG System

To go beyond basic RAG (retrieve then answer) and reach production quality, consider the following.

1. Prompt Engineering:

   • Assign a role: Give the LLM a clear persona, e.g. "You are a professional financial analyst."
   • Require citations: Add constraints such as "Always cite source documents in your answer, and do not guess when the source is unclear."

2. Hybrid Search:

   • Keyword + vector: Pure vector search (semantic search) alone can be inaccurate. Combine BM25 (keyword-based) search with vector search (semantic) to maximize retrieval accuracy. (Supported by most advanced search engines.)

3. Re-ranking:

   • Instead of blindly using the top $K$ retrieved documents (e.g., 10), use a separate reranker model to re-order them. Using only the most relevant top 3–5 for final answer generation dramatically improves answer quality.

Follow these steps and you can go beyond a simple Q&A chatbot to a powerful AI retrieval-augmented generation (RAG) system that leverages your company's knowledge base.