Prompting vs RAG vs fine-tuning + Fine-tuning methods
Comes up in: AI engineering, AI system design
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- decision framework (knowledge → RAG
- format/style/behaviour → fine-tuning or prompting
- latency/cost → smaller fine-tuned model)
- data requirements
- evaluation before and after
- distillation
- supervised fine-tuning
- LoRA/QLoRA (cite Hu et al. 2021
- Dettmers et al. 2023)
- preference tuning (DPO) at concept level
- dataset preparation
- overfitting
- hosted fine-tuning APIs vs self-hosted
Words you will know
What you will practise
- decide for 6 scenarios
- LoRA fine-tune a small model on a notebook (optional, free GPU tier)
Interview questions this answers
- When would you fine-tune instead of using RAG or better prompts?
- What is LoRA and why is it cheaper than full fine-tuning?
- When would you run a model locally or self-host?