Research · v1.1 · published 18 Aug 2026 · updated 23 Aug 2026 · CC BY 4.0
Production AI Benchmark 2026.
What five shipped AI systems actually cost and score in production: RAG queries at $0.006–0.02, regulatory monitoring at ~$0.02 per weekly run, agent evals lifted from 59% to 72.4% by closed-loop optimization, and 1,609 automated tests across the fleet. First-party data, measured by the team that operates the systems — published as an open dataset.
Quotable findings
Each finding is a single sourced sentence, written to be quoted as-is. All figures are first-party measurements from systems starmo built and operates; the full dataset is downloadable below.
A production retrieval-augmented generation query costs $0.006–0.02 including hybrid retrieval, LLM reranking, generation, and a retrieval-sufficiency check — measured by per-query tracing in a live aviation assistant (starmo, 2026).
DocsAI · per-query cost tracing in production
Automated regulatory change-monitoring across four scraper architectures costs about $0.02 per weekly run in LLM spend, because summaries are generated only for sections that actually changed (starmo, 2026).
Regulation Monitoring · airline production module
Closed-loop prompt-and-tool-description optimization lifted an agent's tool-selection eval score from 59% to 72.4% over roughly twenty accept/reject-gated iterations, graded by a deterministic rubric rather than an LLM judge (starmo, 2026).
docsai-autoresearch · 28-case eval vs mocked production API
Five production AI systems shipped by a two-person team carry 1,609 automated backend tests between them — 630, 755, 173, and 51 per system — counted from source, not estimated (starmo, 2026).
All systems · test functions counted in source
A live regulatory-monitoring SaaS tracks 5,048 EASA regulations across 2,764 aircraft types — ingested on a six-hour cycle, so the live count only grows (starmo RegNotify, count on 2026-08-18).
RegNotify · live at regnotify.com
A single-pass AI generation pipeline produced 7,453 public SEO URLs — page content, RAG context, and FAQ structured data generated together per document (starmo RegNotify, 2026).
RegNotify · AI-as-SEO pipeline
Postgres with pgvector (HNSW) served retrieval in all five production systems; no dedicated vector database was required at business-corpus scale (starmo, 2026).
Cross-system architecture finding
A 100-case golden evaluation set with LLM-as-judge scoring guards releases of a production retrieval assistant against quality regressions (starmo, 2026).
DocsAI · release gate
Methodology — read this before citing
This is operator-reported, first-party data from five systems built and run by one team: two live SaaS products (RegNotify, nowreply), two client production systems in aviation, and one internal agent system. n=5. It is not a survey, the ranges are not confidence intervals, and results from one team's architecture choices may not transfer to yours. We publish it because most AI cost and quality numbers in circulation are estimates by people who don't operate the systems — these are measurements, with their limits stated. Counts (tests, migrations, corpus sizes) are taken from source code and live systems on the stated date; costs come from per-query and per-run tracing built into the systems themselves.
Download the dataset
CSV · JSON — CC BY 4.0, cite as:
APA: Tinjak, R. (2026). Production AI Benchmark 2026: costs, tests, and eval scores from five shipped systems. starmo.it. https://starmo.it/reports/production-ai-benchmark-2026
BibTeX: @techreport{starmo2026benchmark, author={Tinjak, Rijad}, title={Production AI Benchmark 2026}, institution={starmo.it}, year={2026}, url={https://starmo.it/reports/production-ai-benchmark-2026}}
Questions this report answers
- What does a production RAG query cost in 2026?
- In starmo's production system, $0.006–0.02 per query including hybrid retrieval, LLM reranking, generation, and a sufficiency check — measured with per-query tracing across live traffic. Costs concentrate in reranking and generation.
- Do you need a dedicated vector database for production RAG?
- In this dataset, no: Postgres with pgvector (HNSW indexes, hybrid dense + full-text search) served retrieval in all five production systems. Dedicated vector stores earn their place at scales most business corpora don't reach.
- How much can automated prompt optimization improve an AI agent?
- In starmo's closed-loop harness: tool-selection eval scores rose from 59% to 72.4% over ~20 iterations, each accepted or rejected against a deterministic rubric. Small eval set (28 cases), one tool family — a real but bounded result.
- How many tests does a production AI system need?
- starmo's five systems carry 1,609 automated tests total — roughly 120–760 per system depending on scope. The consistent pattern: test counts scale with the system's blast radius, and evaluation sets (golden questions) complement unit tests rather than replace them.
- Where does this data come from?
- First-party measurements from five AI systems starmo built and operates — two live SaaS products, two client production systems, and one internal agent system. n=5 systems, one team, operator-reported. It's published as an open CC BY 4.0 dataset precisely so it can be checked and cited.
Changelog — v1.1, 23 Aug 2026: report moved to its permanent home at starmo.it; content unchanged. v1.0, 18 Aug 2026: initial publication. Updates will be listed here.