Give us an endpoint and key. We run our reference corpus, verify every extracted number deterministically, and let you diff runs after each model or prompt change.
EN v1.1 pilot · 5-document Free smoke · your endpoint and key.Base URL, model name, auth header. OpenAI-compatible chat completions is all we need. The key stays in memory for the run and is never written to a database or log.
Call run_benchmark from your agent over MCP, or POST /v1/runs from CI. Designed for self-serve use once available. The planned set prompt and format are fixed, so runs will be comparable.
Per-field accuracy, error classes, latency, tokens, cost. Rerun after a change and the diff tells you what got fixed, what regressed, and whether it beats the noise band.
run_benchmark(
set_id="en-fin-statements-v1-1-dev",
target={provider, base_url, model, auth_header_value},
confirm_document_transfer=true,
idempotency_key="replace-with-unique-run-id"
)
→ get_report(run_id) · diff_runs(a, b)
curl -X POST https://bench.cz-agents.dev/v1/runs \
-H "Authorization: Bearer $BENCH_TOKEN" \
-H "Content-Type: application/json" \
-d '{"set_id":"en-fin-statements-v1-1-dev",
"target":{"provider":"your-provider",
"base_url":"https://your-provider.example/v1",
"model":"your-model",
"auth_header_value":"Bearer YOUR_PROVIDER_KEY"},
"confirm_document_transfer":true,
"idempotency_key":"replace-with-unique-run-id"}'
# fail the build if diff regresses beyond the noise band
Examples use the EN DEV set. Free tokens use en-smoke-v1.
Every wrong value is classed, so you know whether to fix the prompt, the parser or the model. Scores come with a noise band from repeated baseline runs — a change only counts when it is bigger than the noise.
15 US large-cap nonfinancial SEC 10-K filings · 164 scored cells. Visible statement tables only; not scanned PDFs, Companies House or a broad English corpus.
Terms and amounts from public SEC exhibits.
precision@k, per-class accuracy, chrF — same report envelope.
Prices in USD. Inference runs on your endpoint, on your key, at your cost. The available financial-statement corpus is the EN v1.1 pilot.
5 documents. Checks connectivity, response format and latency. No accuracy score.
Run smoke testUp to 30 DEV runs within 30 days. EN DEV subset, reference values visible, per-cell diff.
Buy dev pack20 full EN runs per billing month, including the blind holdout. Regression diff with noise band.
SubscribeA reference set built on your document class, or the whole stack in your VPC.
UnavailableSource, licence, attribution, PII status, third-party content, version and prompt hash — in the API and every report.
Decimal arithmetic, no LLM-as-judge. A reference model's score on the same set version tells your endpoint apart from the harness.
Per-field aggregates only. References cannot be reconstructed by running the set again and again.
Every set is prepared with a human-verified sample of reference values, checked against the source page and flagged in the manifest.
We don't receive any of your documents. Your key is held in memory for the run only. A run fails only on transport errors (auth, timeout, provider outage) and returns a reason with no partial score. A malformed model answer is scored as a model error, not hidden.
Anything that speaks OpenAI-compatible chat completions: hosted providers, gateways and routers, self-hosted models behind vLLM.
Each expected field is classed exact, wrong, missing or invalid; wrong values are classed further as scale, sign, period, fabricated or null. No model in the loop.
Showing cells would leak the references through repeated runs, and the score would stop meaning anything.
They are good for comparing models, prompts and routes and catching regressions. They are not proof of accuracy on your document class — that is what a custom set is for.
A versioned set on your document class with the same manifest, holdout split and scorer — optionally the whole stack inside your VPC. From $12,000.
EN v1.1 pilot · 5-document Free smoke · your endpoint and key.