Artificial Analysis, the independent evaluation group behind benchmarks like GDPval-AA and AA-Briefcase, has launched Optima, a platform that lets teams build AI benchmarks from their own data instead of relying on public leaderboards. According to The Decoder, Optima is available now and scores models not just on quality but on cost per task and time per task.

The idea targets a gap that most AI teams already know from experience: a model that tops a general coding or reasoning leaderboard is not automatically the right pick for extracting line items from invoices, matching a law firm's writing style, or running a finance agent at scale. Building a custom eval pipeline to test that has historically meant weeks of engineering work most teams skip, which is exactly why they end up choosing models by leaderboard rank instead of by fit.

Three ways to feed it your own data

Optima accepts several kinds of source material. Users can upload existing evaluation datasets from their own files or from Hugging Face, or import AI agent traces directly from observability platforms like Arize, Braintrust, and Langfuse. Developers can also install a skill that pulls context from their coding environment and past sessions.

Teams without ready-made eval data can instead describe the use case and provide sample inputs and outputs; Optima generates suggested test inputs, evaluation criteria, and example tasks that users review and refine before running the benchmark. Scoring runs one of two ways: rubric-based grading against explicit criteria, or pairwise comparison, where users rank a sample of response pairs and Optima extrapolates a full ranking across the dataset, the same method Artificial Analysis already uses for GDPval-AA and AA-Briefcase.

Cost and latency as first-class metrics

The more distinctive design choice is treating cost per task and time per task as standalone comparison axes rather than footnotes to a quality score. That matters most for agentic workloads, where raw token pricing is a poor proxy for total cost: a cheaper model that needs more retries, fails more often, or produces output that requires manual cleanup can end up costing more per completed task than a pricier one that gets it right the first time.

• Early testers reportedly built finance and accounting agent benchmarks to find models that cut cost by a factor of ten without a meaningful quality drop.
• Others tested which model best matched the writing style of lawyers, or most accurately tagged elements in a proprietary image dataset.

Pricing follows the same usage-based logic: Optima charges token costs with no markup, plus $0.125 per criterion per model for rubric evaluations and $0.375 per pairwise comparison. The platform holds an estimated balance at benchmark creation, at each run, and at each evaluation round, then settles against actual usage.

What custom benchmarks don't fix

Optima closes the gap between generic leaderboards and a specific workflow, but it doesn't touch the deeper methodological problems that make benchmarks unreliable in the first place. An Epoch AI analysis found that prompt wording and temperature settings alone can shift a model's score on the same benchmark, and that simply swapping the agent scaffold on SWE-bench-style tests accounts for up to 15 percentage points of difference. A separate review of 445 benchmark papers from major AI conferences found that nearly all had at least one methodological weakness, including vague definitions, unrepresentative samples, or no statistical validation, and that only about one in ten used complete, real-world tasks.

None of that goes away just because the benchmark is now yours. A custom test built on a badly chosen sample or a loosely worded rubric is still a badly designed test — Optima gives builders the tooling to run it, not the judgment to design it well.

AiiN's takeaway

For teams that already have agent traces sitting in Arize, Braintrust, or Langfuse, or eval datasets gathering dust on Hugging Face, Optima is a low-effort way to turn that data into a model comparison without standing up a separate eval stack — the pay-per-use pricing means there's no upfront cost to trying it on a single workflow. Builders who don't trust their own ability to write a rigorous rubric are likely better served by the pairwise mode, since it only requires judging examples rather than defining criteria up front.

The metric that still won't show up on the dashboard, in our estimation, is whether the cheapest, fastest model is actually worth deploying — a fast, inexpensive output that needs heavy human rework can be a net loss even when it wins on every number Optima reports. Cost and time per task are a real improvement over quality scores alone, but they're still proxies, not a verdict on business value.