[Complete Code Example Guide] Building Your Own RAG System with LangChain & LlamaIndex

[Complete Code Example Guide] Building Your Own RAG System with LangChain & LlamaIndex

"LLMs are all-powerful? Not quite. But to use an LLM well, you need a knowledge base."

The pace of LLM (large language model) progress over the past few years has been remarkable. They can seem almost magical at answering complex questions, writing, and coding. But as developers, when we try to put this powerful tool into a real product, we often hit a wall.

"What if the model's knowledge isn't up to date?" "What if it doesn't know a specific policy in our internal company manuals?" "Won't it sometimes make things up (hallucination)?"

These problems are exactly why RAG (Retrieval-Augmented Generation) is needed. RAG is the process of having an LLM answer by consulting an external, trustworthy knowledge base.

Theory alone is not enough. Today I will turn the abstract idea of RAG into working code blocks and a step-by-step workflow—a complete blueprint you can follow today and actually produce a result.

📚 Step 1: Data Preparation and Embedding (Building the Knowledge Base)

The success of a RAG system depends 80% on how well you build this knowledge base. Think of it as classifying books in a library and organizing them so they are easy to search.

We start with unstructured documents such as PDFs and DOCX files. The first goal is to convert those documents into numeric vectors the LLM can work with.

🧱 Workflow Overview: Load $\rightarrow$ Split $\rightarrow$ Embed $\rightarrow$ Store

1. Document Loader: Load files of various formats (PDF, TXT, etc.).
2. Text Splitter: Split the loaded documents into appropriately sized chunks that fit the LLM's context window. Too large and you get noise; too small and you lose context.
3. Embedding: Convert each chunk into a high-dimensional vector (an array of numbers) using an embedding model (e.g., OpenAI text-embedding-ada-002).
4. Vector Store: Store these vectors in a specialized database (Vector DB) that is easy to search.

💻 Hands-on Code: Loading Data and Vectorizing (LlamaIndex Example)

In a real environment, you need to install libraries first, e.g. pip install llama-index pypdf chromadb openai.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb
import os

# 1. ( PDF set)
# os.makedirs("data", exist_ok=True)
# print("💡 'data' PDF file .")

# 2. Document Loader:
print("✅ 1. when...")
try:
    documents = SimpleDirectoryReader("data").load_data()
 print(f"✨ {len(documents)} .")
except FileNotFoundError:
 print("🚨 'data' . file .")
    exit()

# 3. Vector Store Index
# ChromaDB , DB .
print("✅ 2. ChromaDB Index when...")
db = chromadb.EphemeralClient # when
chroma_collection = db.get_or_create_collection("company_manual_index")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)

# 4. Index ( , , saveto )
# LlamaIndex .
index = VectorStoreIndex.from_documents(
    documents,
    vector_store=vector_store,
    show_progress=True
)

print("\n🎉 ! .")

💡 Developer debugging tip: If you hit an Embedding Model related error, first check that your API key (OPENAI_API_KEY, etc.) is set correctly in the environment variables, and that the model is actually available (e.g., whether you have a paid API).

🚀 Step 2: Building Orchestration (Completing the Q&A Pipeline)

Once the knowledge base is ready, you need to build the "brain" that takes a user question and generates an answer. That process is orchestration.

Core flow: $$\text{User Query} \xrightarrow{\text{Embedding}} \text{Vector Search} \xrightarrow{\text{Retrieval}} \text{Context Documents} \xrightarrow{\text{Prompt Augmentation}} \text{LLM} \xrightarrow{\text{Answer}} \text{Final Answer}$$

📊 LangChain vs. LlamaIndex: Which Tool Is Right for You?

Both frameworks are well suited to RAG, but they differ in philosophy.

Bottom line: Both are excellent for laying the foundation of a RAG system, but if you are focused purely on document-based knowledge retrieval, LlamaIndex can be more intuitive.

💻 Code Example (LangChain-based)

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.chains import RetrievalQA

# 1. save ( DB )
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(
 documents=loaded_documents, #
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# 2. (Retriever) settings
retriever = vectorstore.as_retriever(search_kwargs={"k": 3}) # 3

# 3. QA
llm = ChatOpenAI(model_name="gpt-4o")
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
 return_source_documents=True #
)

# 4. run
query = " when ?"
result = qa_chain({"query": query})

print(f"🤖 : {result['result']}")
print("\n📚 :")
for doc in result['source_documents']:
    print(f"- {doc.metadata.get('source')}: {doc.page_content[:50]}...")

💡 Advanced Topics: Raising the Quality of Your RAG System

To go beyond basic RAG (retrieve then answer) and reach production quality, consider the following.

1. Prompt Engineering:

   • Assign a role: Give the LLM a clear persona, e.g. "You are a professional financial analyst."
   • Require citations: Add constraints such as "Always cite source documents in your answer, and do not guess when the source is unclear."

2. Hybrid Search:

   • Keyword + vector: Pure vector search (semantic search) alone can be inaccurate. Combine BM25 (keyword-based) search with vector search (semantic) to maximize retrieval accuracy. (Supported by most advanced search engines.)

3. Re-ranking:

   • Instead of blindly using the top $K$ retrieved documents (e.g., 10), use a separate reranker model to re-order them. Using only the most relevant top 3–5 for final answer generation dramatically improves answer quality.

Follow these steps and you can go beyond a simple Q&A chatbot to a powerful AI retrieval-augmented generation (RAG) system that leverages your company's knowledge base.