Run log · example data
One template, from first draft to production.
A worked example: one invoice extractor, four weeks, every screen along the way. The numbers are illustrative. Every step is something BrightEngine does today.
-
01
Draft
Write the template.
Pick a model, write the prompts, and define the JSON shape the answer must follow. Placeholders like
{{vendor_hint}}pull in context from each case. Common prompts and schemas can come from your presets library.Some jobs are better asked as typed questions. A System One version asks Choice, Noul (a probability from 0 to 1) and Score questions through TypeSafe’s Jev model, and keeps the full probability evidence next to each answer.
Invoice Data Extractor
v1 · DraftCategory Invoices
System Prompt
You extract structured data from supplier invoices. Return the total exactly as printed. List every line item with its quantity.
User Prompt
Extract the invoice fields from the document below. The supplier is probably {{vendor_hint}}.
Structured Output JSON SchemaInvoiceFields
{ "type": "object", "properties": { "invoice_number": { "type": "string" }, "invoice_date": { "type": "string", "format": "date" }, "vendor_name": { "type": "string" }, "total_amount": { "type": "number" }, "line_items": { "type": "array", "items": { … } }, "payment_terms": { "type": "string" } } }Model configuration
- Model
- gpt-6.1-sol
- Provider
- OpenAI · direct
- Thinking level
- Model default
- Temperature
- 0.0 Precise
Validation keys
- invoice_number
- invoice_date
- vendor_name
- total_amount
- line_items
- payment_terms
The fields that count toward the score.
Models and providers
One picker for every version
Source What you bring Example models Direct OpenAI, Anthropic or Google key gpt-6.1-sol, claude-opus-5-5, gemini-3.8-flash OpenRouter Team catalog with thinking levels and web search x-ai/grok-4.6, z-ai/glm-5.3-flash System One TypeSafe key for typed decision models jev-1.13.0 Custom An endpoint you host, tested from settings your-org/invoice-ft-3 Keys can belong to the whole team or to one person. A template tells you before you run it if its model’s key is missing.
-
02
Cases
Collect evaluation cases.
An evaluation case is a real document plus the answer you expect. Upload PDFs and images or link to a file. Industry-leading document AI providers turn each one into Markdown the model can read, and you can check the result next to the source.
Most fields are checked by direct comparison. Numbers can use an acceptance rule, such as equal to, at least, at most or between. Free-text answers, such as payment terms, can go to an LLM judge instead.
Invoices
96 cases
Name Content Invoice with handwritten total INV-20417.pdfMarkdownCase State Two-page invoice with credit note INV-20388.pdfMarkdown Scanned receipt, rotated 90° scan_0412.jpgMarkdown Invoice in NOK with Norwegian labels faktura_8812.pdfMarkdown Supplier statement, three invoices files.example.com/stmt-q3Processing Showing 5 of 96 cases · example data · converted to text by industry-leading document AI providers
Edit Case
Invoice with handwritten total
Case State
vendor_hint
Nordic Supplies AS
Expected Answer
Mode Direct comparison
- invoice_number
- Text
- INV-20417
- invoice_date
- Text
- 2026-09-28
- total_amount
- Number
- 12480.00 · Between 12479.50 and 12480.50
- line_items
- List
- 4 items
- payment_terms
- LLM judge
- Required points: states when payment is due, gives the term in days
-
03
Evaluation · 62%
The first evaluation passes 60 of 96 cases.
Every field in every answer gets its own verdict: Pass, Fail, Error or Not scored. That is how you learn that invoice numbers and dates are already solid and totals and line items are not. The scopes take you straight to the problems.
Try it: pick a scope, then pick any verdict to see the expected and actual values behind it.
Evaluation results · v1 · 96 cases
-
04
→
Evaluation · 91%
Iterate until the numbers hold.
Each change is a new version, so nothing is lost and any version can be made active again. Template Comparison shows exactly what changed. Compare with baseline lists the cases that improved, regressed or stayed the same.
When the model is right and your answer key is wrong, overwrite the expected value with the actual one. When a verdict needs a human call, override it, and the automated verdict stays on record. Rerun any evaluation, or just its failures, and BrightEngine remembers where each rerun came from.
Evaluations · Invoice Data Extractor v1 to v6 · 62% to 91%
Compare templates
v5 (Published) → v6 (editing)
Settings
Setting v5 · Published v6 · Editing Model x-ai/grok-4.6 x-ai/grok-4.6 Thinking level Low Low Temperature 0.0 0.0 User prompt
v5 · Published
1Extract the invoice fields from {{content}}.
2Removed: Use the printed total.
3List every line item.
v6 · Editing
1Extract the invoice fields from {{content}}.
2Added: When the total is handwritten,
3Added: prefer it over the printed one.
4List every line item.
-
05
Promote
Promote the tested version to production.
Promotion copies the content of v6 onto the production template as a new version, v12, and can make it active straight away. The production template keeps its GUID, so anything already calling it picks up the new version without a code change. Development and production can live in separate teams.
v6 promoted to Production / Invoice Data Extractor as v12 · GUID unchanged
-
06
Live
Call it from your workflow.
Your ERP, inbox automation or back-office script posts a document to the template’s trigger URL. BrightEngine answers 202 straight away, runs the active version, and posts the result to your callback URL.
Add a client request ID and the callback arrives in a signed envelope. Verify it with your team’s webhook secret and you know the result came from BrightEngine and was not replayed.
Trigger URL
Use this URL to run this template from external sources.
https://app.brightengine.example/api/trigger/3f9b2c1e-8a47-4d0e-b6a2-91c0e5d7fd41Your system → BrightEngine
curl -X POST \ $BRIGHTENGINE_URL/api/trigger/3f9b2c1e-8a47-4d0e-b6a2-91c0e5d7fd41 \ -H "Content-Type: application/json" \ -d '{ "documents": [{ "url": "https://erp.example.com/inbox/INV-20931.pdf", "title": "INV-20931" }], "data": "Supplier hint: Nordic Supplies AS", "callback_url": "https://erp.example.com/hooks/brightengine", "client_request_id": "erp-INV-20931" }'Response202 Accepted
{ "batch_id": 48213, "status": "accepted" }BrightEngine → your callback
X-Brightengine-Timestamp: 1760434262 X-Brightengine-Signature: sha256=9f2c41…e07b { "schema_version": "1.0", "client_request_id": "erp-INV-20931", "job_id": "48213", "status": "success", "result": "{\"invoice_number\":\"INV-20931\", \"total_amount\":12480.0, …}", "error": null }Verify the signature · Node.js
// HMAC-SHA256 over method, path, timestamp and raw body const expected = createHmac("sha256", secret) .update(`POST\n${path}\n${timestamp}\n${rawBody}`) .digest("hex"); const valid = timingSafeEqual( Buffer.from(`sha256=${expected}`), Buffer.from(signature), ); // Reject timestamps older than 300 seconds. -
07
Capture
Production calls become new cases.
Every call lands in the API call log with its input, output, status, duration and cost. When an answer is right, capture it as an evaluation case in one click. Capturing says the answer is correct, so BrightEngine never does it on its own unless you tell it to.
A capture policy picks candidates for you. In review mode it sends matching calls to a review queue. In automatic mode, which you switch on deliberately, it captures them straight away. Try the queue below.
API Call Logs · Capture Policy · Review queue
-
08
Experiment · live
Test the next version on real traffic.
A cheaper model looks promising. A comparison experiment runs it alongside production on real calls, and callers still get the production answer. Run it live for up to 30 days, or against up to 500 past calls.
Set success criteria before it starts and BrightEngine reports Criteria met, Criteria not met or Inconclusive. With no criteria, you get the evidence without a verdict. Keep any interesting samples as new evaluation cases.
Comparison experiment · v12 vs v13 · Criteria met
Platform
Everything around the loop.
What the steps above didn’t show: the parts you need once testing before production becomes a habit.
- Cost on every run
- Input, output and cached tokens with a dollar cost for every run and every judge call, as billed by the provider where available and estimated otherwise.
- Teams and roles
- Admins, Editors and Members. Keep development and production in separate teams and link them with a promotion target.
- Typed questions with System One
- Ask Choice, Noul (a probability from 0 to 1) and Score questions through TypeSafe’s Jev models. Each answer arrives with its full probability distribution as model evidence.
- Locked templates
- Lock a template to prevent unintended changes. Only team admins can unlock it.
REST API · v1
Everything in the app has an endpoint.
Automate the whole loop: create templates and cases, submit evaluations, read results and trigger production runs. Long-running work returns an operation you can poll or receive by callback.
| Endpoint | What it covers |
|---|---|
| /api/v1/templates | Create, version, activate, lock, promote, read API logs |
| /api/v1/evaluation-cases | Create in batches, bulk update, convert documents to Markdown |
| /api/v1/evaluations | Submit, read results and runs, rerun, override a verdict |
| /api/v1/operations/{id} | Poll the status of queued work |
| /api/v1/team | Members, invitations, API tokens, provider keys, custom models |
| /api/trigger/{guid} | Run the active version of a template from production |
- Scoped API tokens
- Tokens belong to a team and carry only the abilities you grant. Each token is shown once, when you create it.
- templates:read · templates:write · evaluation-cases:write · evaluations:read · evaluations:write · provider-keys:manage · api-tokens:manage
- Webhook signing secret
- One active secret per team signs every callback with HMAC-SHA256 over the method, path, timestamp and body. Rotate it from team settings.
Start with the examples you already have.
Bring ten real examples and the answers you expect. By the end of the first evaluation you will know where your template stands, field by field.