LLM Performance Improvement Series Part 2: A Practical Guide to LoRA and QLoRA Fine-Tuning

Beyond RAG: When Fine-Tuning Is Needed

In Part 1, we covered how to build a reliable in-house chatbot with RAG. RAG is fast and flexible, but when a model needs to deeply internalize a domain’s language style and patterns, fine-tuning is the answer.

RAG vs. Fine-Tuning: How to Choose

LoRA: The Core of Efficient Fine-Tuning

LoRA (Low-Rank Adaptation) freezes the original weights and trains only low-rank decomposition matrices. It cuts the number of trainable parameters by 100× or more while approaching full fine-tuning performance.

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
 r=16, # : ·,
 lora_alpha=32, # ( r 2)
 target_modules=["q_proj", "v_proj"], #
    lora_dropout=0.1,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
peft_model = get_peft_model(model, lora_config)
peft_model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 8,030,261,248 || 0.052%

QLoRA: Fine-Tuning on a Consumer GPU

QLoRA combines 4-bit quantization with LoRA so you can fine-tune a 70B model on 24GB of VRAM.

from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
 bnb_4bit_quant_type="nf4", # NormalFloat4: 4bit
    bnb_4bit_compute_dtype=torch.bfloat16,
 bnb_4bit_use_double_quant=True, #
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto",
)

Preparing the Dataset: Quality Beats Quantity

The most important factor in fine-tuning is data quality.

# instruction-response
dataset_example = {
 "instruction": " .",
 "input": " .",
 "output": " . check . , when ."
}

#
checklist = [
 "100~500 reply ",
 " ",
 " ",
 " ",
 " ",
]

Data volume guide:

• Style tuning: 100–500 examples
• Domain specialization: 1,000–5,000 examples
• Learning a new task: 10,000+ examples

Training Configuration

from transformers import TrainingArguments
from trl import SFTTrainer

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # effective batch = 16
    warmup_ratio=0.03,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = SFTTrainer(
    model=peft_model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=2048,
)
trainer.train()

Post-Training Evaluation

# BLEU, ROUGE — LLM
from evaluate import load

rouge = load("rouge")
scores = rouge.compute(predictions=model_outputs, references=reference_outputs)
print(f"ROUGE-L: {scores['rougeL']:.3f}")

# (: )
def domain_accuracy(outputs, references, domain_terms):
    hits = sum(any(t in o for t in domain_terms) for o in outputs)
    return hits / len(outputs)

In Part 3, we cover strategies for routing and ensembling multiple LLMs to optimize accuracy and cost at the same time.

Training Failure Playbook by Symptom

Pre-Training Checklist

• Is this really a fine-tuning problem — have you documented cases showing prompting/RAG cannot solve it
• Evaluation criteria selected — record baseline scores on the same eval set before training
• Data license and PII — reviewed that training data contains no customer-identifying information
• Experiment tracking — record hyperparameters, data version, and adapter weights together (reproducibility)

Frequently Asked Questions (FAQ)

Q. Should I use LoRA fine-tuning or RAG? A. They serve different purposes. RAG retrieves and injects up-to-date or external knowledge (fact- and document-based). LoRA teaches the model a speaking style, format, and domain expression (style and task adaptation). Use RAG when you need current information; use LoRA when you need a specific format, tone, or classification accuracy.

Q. Can I use LoRA and RAG together? A. Yes — it’s a common combination. Lock in domain tone and output format with LoRA, and inject real-time evidence with RAG, and you get “our company voice + current facts” at once. Just keep the LoRA training data and RAG documents aligned to the same domain.