Running open-weight models locally
Comes up in: AI engineering
This concept is being written. It gets the full five steps — explanation, predict, interview angle, practice with tests, and review cards. Here is exactly what it will cover, so you can study it elsewhere in the meantime.
What it covers
- open-weight model families (licence differences)
- quantisation (GGUF, 4-bit) and quality trade-offs
- Ollama, llama.cpp, LM Studio, vLLM
- hardware needs
- OpenAI-compatible local endpoints
- privacy and offline use cases
Words you will know
What you will practise
- run a local model and call it from your app
Interview questions this answers
- What does quantisation trade off?