The one-line version
Choose the cheapest model that clears your quality bar on your own tasks — not the one with the highest leaderboard score.
The 5-step framework
- Define the task. "Best model" is meaningless without one. Summarization, support, agentic coding, RAG, extraction and creative writing all weight the scoring dimensions differently.
- Set hard constraints. Budget, latency, context and compliance narrow the field fast (details below).
- Shortlist with the right benchmarks. Use category leaderboards, not the overall ranking — coding → SWE-bench, reasoning → GPQA, Chinese → SuperCLUE.
- Run your own evaluation. 50–200 real tasks from your domain, scored blind. This step consistently overturns leaderboard-driven picks.
- Decide, then keep measuring. Models change monthly. Put evaluation on a schedule and track cost per successful task.
Your four hard constraints
Budget
Tokens per request × monthly requests. Output tokens usually cost 2–5× input tokens.
Latency
Interactive products need high tokens/sec and low time-to-first-token. Batch jobs don't.
Context
Entire repositories or long documents need a large window — and good long-context retrieval.
Compliance
Regulated industries may require data residency or private deployment, ruling out hosted APIs.
Benchmarks are a filter, not a verdict — vendors tune to them and harnesses differ. Your evaluation distribution is never identical to the benchmark distribution, which is exactly why step 4 exists.