LLM Fine-Tuning: LoRA, QLoRA, and PEFT in Practice
Fine-tuning is often presented as a solution in search of a problem. Before investing weeks of engineering, understand when fine-tuning actually beats prompt engineering.
When to Fine-Tune vs. Prompt Engineer
Prompt engineering first (almost always correct):
- Task demonstrated in <10 in-context examples
- You need to switch behaviors frequently
- Data volume is <1000 examples
Fine-tuning wins when:
- Consistent output format at scale (cost savings from shorter prompts)
- Domain-specific knowledge not in pretraining
- Task requires >10 few-shot examples
- Latency is critical (smaller fine-tuned model beats larger base model)
- Data privacy prevents sending to API
LoRA: The Core Concept
Full fine-tuning updates billions of parameters. LoRA adds small trainable matrices while freezing the base model:
Original W (4096 x 4096) = 16M parameters
LoRA: W + BA where B (4096 x 16) and A (16 x 4096) = 131K parameters
Trainable: 0.8% of original!
from peft import get_peft_model, LoraConfig, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_name = 'meta-llama/Llama-3.1-8B-Instruct'
model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.bfloat16, device_map='auto'
)
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank: 8-64
lora_alpha=32, # Scaling (usually 2x rank)
lora_dropout=0.1,
target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj',
'gate_proj', 'up_proj', 'down_proj'],
bias='none',
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 83,886,080 || all params: 8,114,114,560 || trainable%: 1.03
QLoRA: Fine-Tune on Consumer GPUs
QLoRA quantizes the base model to 4-bit, enabling fine-tuning a 70B model on a 48GB GPU:
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4', # NormalFloat4 (best quality)
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, # Quantize the quantization constants
)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map='auto'
)
# Prepare model for QLoRA training
from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
# 70B model now fits in 48GB VRAM!
Dataset Preparation
Data quality matters far more than quantity:
from datasets import Dataset
from transformers import AutoTokenizer
def format_instruction(example: dict) -> str:
return f'''<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
{example['instruction']}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
{example['output']}<|eot_id|>'''
def tokenize_dataset(examples, tokenizer, max_length=2048):
formatted = [format_instruction(ex) for ex in examples]
tokenized = tokenizer(
formatted,
truncation=True,
max_length=max_length,
padding=False,
return_tensors=None,
)
# Mask padding tokens in labels
tokenized['labels'] = [
[-100 if t == tokenizer.pad_token_id else t for t in ids]
for ids in tokenized['input_ids']
]
return tokenized
# Data quality checklist:
# - Minimum 500 examples (preferably 1000+)
# - Consistent format
# - No personally identifiable information
# - Balanced distribution of task types
# - Human review on 10% sample
Training Configuration
from transformers import TrainingArguments
from trl import SFTTrainer
training_args = TrainingArguments(
output_dir='./checkpoints',
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size: 16
learning_rate=2e-4,
lr_scheduler_type='cosine',
warmup_ratio=0.03,
fp16=False,
bf16=True,
logging_steps=10,
save_strategy='epoch',
evaluation_strategy='epoch',
load_best_model_at_end=True,
gradient_checkpointing=True, # Save memory at cost of 20% speed
optim='paged_adamw_32bit', # QLoRA optimizer
dataloader_num_workers=4,
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
args=training_args,
max_seq_length=2048,
dataset_text_field='text',
packing=True, # Pack multiple short examples per sequence
)
trainer.train()
Merging and Saving
from peft import PeftModel
# Merge LoRA weights into base model
base_model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(base_model, './checkpoints/best')
merged_model = model.merge_and_unload() # Fuse LoRA into base weights
merged_model.save_pretrained('./fine-tuned-model')
tokenizer.save_pretrained('./fine-tuned-model')
Production Serving with vLLM
pip install vllm
# Serve with OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
--model ./fine-tuned-model \
--served-model-name my-fine-tuned-llama \
--tensor-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.95 \
--quantization awq
# Same OpenAI client works with vLLM
from openai import OpenAI
client = OpenAI(base_url='http://localhost:8000/v1', api_key='token')
response = client.chat.completions.create(
model='my-fine-tuned-llama',
messages=[{'role': 'user', 'content': 'Your prompt here'}],
max_tokens=512
)
Evaluation Framework
from evaluate import load
rouge = load('rouge')
bertscore = load('bertscore')
def evaluate_model(model, tokenizer, test_cases):
predictions = []
references = []
for case in test_cases:
inputs = tokenizer(format_instruction(case), return_tensors='pt')
output = model.generate(**inputs, max_new_tokens=256)
pred = tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
predictions.append(pred)
references.append(case['expected_output'])
rouge_scores = rouge.compute(predictions=predictions, references=references)
print(f'ROUGE-L: {rouge_scores["rougeL"]:.3f}')
return rouge_scores
Fine-tuning delivers real value when you have high-quality training data, a specific consistent task, and a reason to move beyond prompt engineering. Start with QLoRA on a small model, validate with evaluation, then scale up.