Bench by cz-agents.dev
MCP + REST deterministic scoring blind holdout

Benchmark AI on real-world documents.

Compare model and route configurations on a versioned reference set. See field-level scores and what changes between runs.

Available now: SEC 10-K (English), Czech pilot (35 balance sheets), synthetic journey-log imports and synthetic insolvency claim–schedule pairs. Your OpenAI-compatible endpoint and API key. Our reference documents. No upload of your confidential documents required.
Run a free smoke test See how scoring works
Illustrative report layout
Numeric exact
[73.5]%
2 baseline runs: [72.4] / [74.6]%
vs last run
+[n] / −[n]
fixed / regressed
fieldverdict
total_assetsexact
revenuewrong · ×1000
bank_loansabsent · correct
result_after_taxwrong · sign
Illustrative values; your report uses the selected set and prompt version.
0 uploads
of your confidential documents — we send our reference set to your endpoint
0 LLM judges
deterministic Decimal scoring
1 call / document
on your endpoint, your key
Blind holdout
aggregates only, never cells
How it works

Three steps. Your endpoint, our reference set.

Our sample statementssent to your AI
Your AI reads themand returns the numbers
Scores against reference values
74%
You see your scoreand exactly where it fails
Change something, run again — see what got better or worse.

1 · Register your endpoint

Base URL, model name, auth header. OpenAI-compatible chat completions is all we need. The key stays in memory for the run and is never written to a database or log.

2 · Start a run yourself

Call run_benchmark from your agent over MCP, or POST /v1/runs from CI. Designed for self-serve use once available. The planned set prompt and format are fixed, so runs will be comparable.

3 · You get a report and a diff

Per-field accuracy, error classes, latency, tokens, cost. Rerun after a change and the diff tells you what got fixed, what regressed, and whether it beats the noise band.

From your agent (MCP)
run_benchmark(
  set_id="en-fin-statements-v1-1-dev",
  target={provider, base_url, model, auth_header_value},
  confirm_document_transfer=true,
  idempotency_key="replace-with-unique-run-id"
)
→ get_report(run_id) · diff_runs(a, b)
From CI (REST)
curl -X POST https://bench.cz-agents.dev/v1/runs \
  -H "Authorization: Bearer $BENCH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"set_id":"en-fin-statements-v1-1-dev",
       "target":{"provider":"your-provider",
                 "base_url":"https://your-provider.example/v1",
                 "model":"your-model",
                 "auth_header_value":"Bearer YOUR_PROVIDER_KEY"},
       "confirm_document_transfer":true,
       "idempotency_key":"replace-with-unique-run-id"}'
# fail the build if diff regresses beyond the noise band

Examples use the EN DEV set. Free tokens use en-smoke-v1.

What you get

Not one number. The reasons behind it.

Every wrong value is classed, so you know whether to fix the prompt, the parser or the model. Scores come with a noise band from repeated baseline runs — a change only counts when it is bigger than the noise.

You send: endpoint URL, model, auth header, optional prices
You never send: documents, ground truth or prompts
JSON or Markdown, provenance and reference-model score in the footer
Error classesillustrative
×1000 scale slip[n]
Wrong line / value[n]
Missing value[n]
Fabricated where absent[n]
Sign / period[n]
Available sets

Start with the first public set.

Available now

SEC 10-K financial statements (English, pilot v1.1)

15 US large-cap nonfinancial SEC 10-K filings · 164 scored cells. Visible statement tables only; not scanned PDFs, Companies House or a broad English corpus.

Available now

Czech balance sheets (pilot)

35 Czech balance sheets · 906 scored cells · 26 field names.

All holdout. Available on Blind and Team only.

Available now

Journey-log imports (synthetic)

30 simulated CSV/XLSX exports · 54 records · 378 scored fields. Pre-extracted text is sent to your endpoint; this set does not test native spreadsheet or PDF ingestion.

12 DEV documents / 18 blind holdout documents · strict raw output scoring. Five-document Free smoke has no score.

Available now

Fictional insolvency claim–schedule pairs (synthetic)

20 operator-owned fictional pairs · 10 DEV / 10 holdout. Scoring checks exact amount strings and the match result, plus the reason null/nonempty contract and an evidence substring.

Five DEV pairs are available in the Free smoke with no score. Reason and evidence checks do not verify explanation correctness or evidential support; no real people or filings are included.

Coming next

Contracts

Terms and amounts from public SEC exhibits.

Coming next

Matching, classification, translation

precision@k, per-class accuracy, chrF — same report envelope.

Pricing

Pay for the reference set, not for compute.

Prices in USD. Inference runs on your endpoint, on your key, at your cost. Available financial-statement pilots: EN v1.1 and Czech (35 balance sheets).

Free smoke $0

5 documents. Checks connectivity, response format and latency. No accuracy score.

Run smoke test
Dev pack $199 one-off

Up to 30 DEV runs within 30 days. EN DEV subset, reference values visible, per-cell diff.

Buy dev pack
Blind setRecommended
$490 / month

20 full EN runs per billing month, including the blind holdout. Regression diff with noise band.

Subscribe
Team $1,990 / month

100 EN runs per billing month, shared across five API keys.

Subscribe
Custom set from $12k

A reference set built on your document class, or the whole stack in your VPC.

Unavailable
Provenance and trust

Know exactly what you are measured against.

Manifest per set

Source, licence, attribution, PII status, third-party content, version and prompt hash — in the API and every report.

Deterministic scoring

Decimal arithmetic, no LLM-as-judge. A reference model's score on the same set version tells your endpoint apart from the harness.

Blind holdout

Per-field aggregates only. References cannot be reconstructed by running the set again and again.

Manifest provenance

Source, licence, attribution, PII status, version and prompt hash are recorded in the set manifest.

FAQ

What happens to my data?

Bench sends our reference documents to your endpoint. No upload of confidential documents is required. Your key is held in memory for the run only. A run fails only on transport errors (auth, timeout, provider outage) and returns a reason with no partial score. A malformed model answer is scored as a model error, not hidden.

Which models and endpoints can I test?

Anything that speaks OpenAI-compatible chat completions: hosted providers, gateways and routers, self-hosted models behind vLLM.

How is scoring done?

Each expected field is classed exact, wrong, missing or invalid; wrong values are classed further as scale, sign, period, fabricated or null. No model in the loop.

Why only aggregates on the holdout?

Showing cells would leak the references through repeated runs, and the score would stop meaning anything.

Are public filings representative of my documents?

They are good for comparing models, prompts and routes and catching regressions. They are not proof of accuracy on your document class — that is what a custom set is for.

What does a custom set involve?

A versioned set on your document class with the same manifest, holdout split and scorer — optionally the whole stack inside your VPC. From $12,000.

Bench early access

EN v1.1 pilot · 5-document Free smoke · your endpoint and key.

Run a free smoke test