- Published on
Choose the right AI model for the job
- Authors
- Name
- Mattia Peretti
- @mattperetti
Model choice is an engineering decision that shows up in both output quality and your bill. Pick a mismatched model and you get worse answers, slower responses, and higher costs. That is true whether you are prompting a chatbot or wiring up an agent workflow. Two resources make the space navigable: models.dev, an open database of specs and pricing, and artificialanalysis.ai, an independent benchmark and cost‑per‑task index. Used together, they turn "what should I run this on?" into a repeatable selection process.
A single glance at price spreads can change your defaults. On models.dev you can see variants from the same lab with very different costs. It lists GPT‑5.6 Sol at 15.00 per million output tokens, while GPT‑5.6 Luna is shown at 4.20. That is a three to four times difference before you generate a single response. If the task does not need Sol's capabilities, you pay a premium for no benefit.
Providers, labs, and frontier models
Who serves you the model is not always who made it. The provider is the API you call to run inference. The provider may or may not manufacture the model. The data model on models.dev makes this separation explicit through Models, Providers, and Labs. Model pages list the providers that serve a given model. Provider pages list every model they make available. Lab pages group canonical models by the authoring team. In practice this means a single model can be available from several providers, and a single provider can serve models authored by different labs. As an example from my own notes, Azure serves Anthropic and Google models without producing any itself.
At the top end of capability you find frontier models. Names like GPT‑5.6 and Claude Fable 5 sit in this tier. artificialanalysis.ai tracks these over time in a view of frontier language model intelligence, and it separates categories like Reasoning versus Non‑Reasoning and Open‑weights versus Proprietary models. That framing helps distinguish "latest" from "strongest at a specific job" and it gives context across 14 or more creators that move at different cadences.
Where to find specs and pricing
I start on models.dev when I need concrete constraints. The homepage presents provider‑agnostic model metadata in a single table. You get model names and IDs, who serves them, context and output limits, support flags for reasoning, tool calls, and structured output, whether the weights are open or closed, and price per million input and output tokens. Release and last updated dates add a recency check. Because the site separates Models, Providers, and Labs, I can click through from a model to its serving options, or from a provider to its full menu, or from a lab to the canonical family, without losing track of who authored what and who will actually handle my API call.
Prices and features here make the quality and cost tension real. Even within a single family you will see multipliers in token costs across variants. That changes not just the steady‑state bill but also how you design retries, tool calls, and structured outputs. For long‑context or structured‑JSON heavy tasks, support flags can be a go or no‑go before I even open a benchmark chart.
Where to find performance and cost per task
Once I have candidates, I switch to artificialanalysis.ai to understand capability and economics at the task level. Its core composite is the Intelligence Index (v4.1), a unified scale built from nine evaluations that include GDPval‑AA v2, τ³‑Banking, Terminal‑Bench v2.1, SciCode, Humanity's Last Exam, GPQA Diamond, CritPt, AA‑Omniscience, and AA‑LCR. Models are compared using first‑party APIs when available, or the median across providers when a first‑party API is not an option. That normalizes differences in serving and puts disparate systems on a shared axis.
The site goes beyond a single score. It plots Intelligence versus Cost per Task, Intelligence versus Time per Task, output tokens per task, output tokens per second, and end‑to‑end latency. There are domain‑specific leaderboards, like coding, image and video generation and editing with Elo scores from blind preference votes, and speech tasks across TTS, STT, and speech‑to‑speech. There is also an API Provider Performance view that compares providers serving the same model on speed and price. These slices make trade‑offs visible. A cheaper token price can be wiped out by longer outputs or higher failure rates, and a fast stream of tokens can still miss a tight SLA if end‑to‑end latency is high.
How I actually compare and choose
The workflow is simple but disciplined. I use models.dev to shortlist by hard constraints and price, then I place those models on artificialanalysis.ai's charts to see where they land on capability, cost per task, and latency. If multiple providers serve the same model, I check the provider performance chart to see who is faster at a given price point. With that view in hand, I choose the first point that clears my capability bar at the lowest cost per task while remaining within my latency budget. Finally, I validate that any tooling or structured output requirements are supported, since these flags on models.dev can quickly disqualify an otherwise strong candidate.
This approach keeps me from defaulting to a familiar brand or the latest headline model when a better‑fit option exists for the job. It also gives me a way to explain and defend the choice, which matters when an agent pipeline spans several tools and the bill winds up in someone else's dashboard.
A chart‑style comparison: GPT‑5.6 Sol, Claude Fable 5, and a Gemini
Putting the method to work, I would compare GPT‑5.6 Sol, Claude Fable 5, and a top Gemini variant in two passes. First, I would anchor capability on artificialanalysis.ai by locating each model on the Intelligence Index and then reading their positions on the Intelligence versus Cost per Task and Intelligence versus Time per Task plots. That gives me a relative map without needing to memorize specific numbers.
Second, I would bring in specs and prices from models.dev to understand economics and fit. GPT‑5.6 Sol is listed with a 1,050,000 token context, a 128,000 token output limit, support for reasoning, tool calls, and structured output, closed weights, and pricing of 15.00 per million output tokens. Claude Fable 5 appears with a 1,000,000 token context, a 128,000 token output limit, and similar support flags. Pricing for this model is not listed on models.dev at the time of writing. For Gemini, models.dev shows multiple variants with very different price profiles. As examples, Gemini 3.1 Pro Preview is shown at 12.00 per million output tokens, while Gemini 3.5 Flash is shown at 1.11. The Intelligence Index helps decide which Gemini counts as "best" for the task, then the models.dev page for that variant fills in the serving, limits, and token rates.
Read together on the intelligence versus cost and time plots, these three points make the trade‑offs obvious: which one reaches your capability bar at the lowest cost per task, and which one meets your latency target. The provider performance view then tells you whether a first‑party API or a third‑party provider is the faster or cheaper way to serve the same underlying model.
Case study: long‑context work with LongCat‑2.0
For tasks dominated by long contexts, I look for concrete limits and prices before anything else. On models.dev, LongCat‑2.0 is listed under the Meituan provider with a 1,000,000 token context and a 131,072 token output limit. It supports reasoning, tool calls, and structured output, has closed weights, and is priced at 2.95 per million output tokens. Those constraints and rates are the starting point for workflows like large document retrieval, long‑form summarization, or multi‑turn planning with big working sets.
After shortlisting it from specs and price, I would position LongCat‑2.0 on artificialanalysis.ai to see how it performs on the Intelligence Index and where it lands on the Intelligence versus Cost per Task and Intelligence versus Time per Task charts. That tells me whether the long‑context advantage holds up once cost per task and latency are included, and whether an alternative with a smaller context but better task‑level economics would be a smarter choice.
Case study: text‑to‑video with Seedance 2.0
For generative video, I want both preference‑based quality and cost clarity. Seedance 2.0 is noted here as a text‑to‑video model, but I could not find it on either models.dev or artificialanalysis.ai at the time of writing. What follows is how I would evaluate it if or when it appears on those sources.
I would start on artificialanalysis.ai's Image and Video leaderboards, specifically the Text‑to‑Video Elo ranking from blind preference votes, and check Seedance 2.0's position relative to the top models on representative prompts. Then I would translate that into economics by using the site's cost‑per‑task and time‑per‑task charts where available for the same model. Finally, I would capture pricing, context, output limits, and any structured output flags from its models.dev page as soon as it is listed, so I can estimate cost per deliverable and set realistic SLAs for renders.
The thread to keep in mind
Both sites make the same point from different angles. models.dev shows that models vary meaningfully in features and price, sometimes by multiples even within a single family. artificialanalysis.ai shows that capability, cost per task, and latency are measurable and often counterintuitive. Together they turn model selection into a defensible choice: pick the model that clears your capability bar at the lowest cost per task and acceptable latency, then verify the provider that serves it fastest at the price you expect.