RAG vs fine-tuning: which one do you actually need?

When a model doesn't know your business, there are two common fixes: retrieval-augmented generation (RAG) and fine-tuning. Teams often jump to fine-tuning because it sounds more powerful. Usually, it's the wrong first move.

What each does

• RAG leaves the model alone and feeds it the right context at query time — store knowledge as searchable chunks, retrieve the relevant ones per request.
• Fine-tuning changes the model's weights by training on your examples — it bakes in tone, format and behaviour.

The rule of thumb

• Reach for RAG when the problem is knowledge: "answer questions about our docs/policies." Facts change; update an index, not the model.
• Reach for fine-tuning when the problem is behaviour: a consistent format, a niche classification, a particular voice.

Most "the model doesn't know X" problems are knowledge problems — which is why RAG solves the majority of real cases, and fine-tuning is often a costly answer to a question nobody asked.

RAG also wins on freshness, traceability (you can cite the source), cost and reversibility. Start there; add fine-tuning only when you've proven a behaviour gap retrieval can't close.

Sources

• Stanford HAI — 2025 AI Index (on model capability and cost trends)