Back to overview

Run log · example data

One template, from first draft to production.

A worked example: one invoice extractor, four weeks, every screen along the way. The numbers are illustrative. Every step is something BrightEngine does today.

  1. 01

    Draft

    Write the template.

    Pick a model, write the prompts, and define the JSON shape the answer must follow. Placeholders like {{vendor_hint}} pull in context from each case. Common prompts and schemas can come from your presets library.

    Some jobs are better asked as typed questions. A System One version asks Choice, Noul (a probability from 0 to 1) and Score questions through TypeSafe’s Jev model, and keeps the full probability evidence next to each answer.

    Invoice Data Extractor

    v1 · Draft

    Category Invoices

    System Prompt

    You extract structured data from supplier invoices. Return the total exactly as printed. List every line item with its quantity.

    User Prompt

    Extract the invoice fields from the document below. The supplier is probably {{vendor_hint}}.

    Structured Output JSON SchemaInvoiceFields

    {
      "type": "object",
      "properties": {
        "invoice_number": { "type": "string" },
        "invoice_date":   { "type": "string", "format": "date" },
        "vendor_name":    { "type": "string" },
        "total_amount":   { "type": "number" },
        "line_items":     { "type": "array", "items": { … } },
        "payment_terms":  { "type": "string" }
      }
    }

    Model configuration

    Model
    gpt-6.1-sol
    Provider
    OpenAI · direct
    Thinking level
    Model default
    Temperature
    0.0 Precise

    Validation keys

    • invoice_number
    • invoice_date
    • vendor_name
    • total_amount
    • line_items
    • payment_terms

    The fields that count toward the score.

    Models and providers

    One picker for every version

    Source What you bring Example models
    DirectOpenAI, Anthropic or Google keygpt-6.1-sol, claude-opus-5-5, gemini-3.8-flash
    OpenRouterTeam catalog with thinking levels and web searchx-ai/grok-4.6, z-ai/glm-5.3-flash
    System OneTypeSafe key for typed decision modelsjev-1.13.0
    CustomAn endpoint you host, tested from settingsyour-org/invoice-ft-3

    Keys can belong to the whole team or to one person. A template tells you before you run it if its model’s key is missing.

  2. 02

    Cases

    Collect evaluation cases.

    An evaluation case is a real document plus the answer you expect. Upload PDFs and images or link to a file. Industry-leading document AI providers turn each one into Markdown the model can read, and you can check the result next to the source.

    Most fields are checked by direct comparison. Numbers can use an acceptance rule, such as equal to, at least, at most or between. Free-text answers, such as payment terms, can go to an LLM judge instead.

    Invoices

    96 cases

    Name Content
    Invoice with handwritten totalINV-20417.pdfMarkdownCase State
    Two-page invoice with credit noteINV-20388.pdfMarkdown
    Scanned receipt, rotated 90°scan_0412.jpgMarkdown
    Invoice in NOK with Norwegian labelsfaktura_8812.pdfMarkdown
    Supplier statement, three invoicesfiles.example.com/stmt-q3Processing

    Showing 5 of 96 cases · example data · converted to text by industry-leading document AI providers

    Edit Case

    Invoice with handwritten total

    Case State

    vendor_hint

    Nordic Supplies AS

    Expected Answer

    Mode Direct comparison

    invoice_number
    Text
    INV-20417
    invoice_date
    Text
    2026-09-28
    total_amount
    Number
    12480.00 · Between 12479.50 and 12480.50
    line_items
    List
    4 items
    payment_terms
    LLM judge
    Required points: states when payment is due, gives the term in days
  3. 03

    Evaluation · 62%

    The first evaluation passes 60 of 96 cases.

    Every field in every answer gets its own verdict: Pass, Fail, Error or Not scored. That is how you learn that invoice numbers and dates are already solid and totals and line items are not. The scopes take you straight to the problems.

    Try it: pick a scope, then pick any verdict to see the expected and actual values behind it.

    Evaluation results · v1 · 96 cases

  4. 04

    →

    Evaluation · 91%

    Iterate until the numbers hold.

    Each change is a new version, so nothing is lost and any version can be made active again. Template Comparison shows exactly what changed. Compare with baseline lists the cases that improved, regressed or stayed the same.

    When the model is right and your answer key is wrong, overwrite the expected value with the actual one. When a verdict needs a human call, override it, and the automated verdict stays on record. Rerun any evaluation, or just its failures, and BrightEngine remembers where each rerun came from.

    Evaluations · Invoice Data Extractor v1 to v6 · 62% to 91%

    Compare templates

    v5 (Published) → v6 (editing)

    Settings

    Setting v5 · Published v6 · Editing
    Modelx-ai/grok-4.6x-ai/grok-4.6
    Thinking levelLowLow
    Temperature0.00.0

    User prompt

    v5 · Published

    1Extract the invoice fields from {{content}}.

    2Removed: Use the printed total.

    3List every line item.

    v6 · Editing

    1Extract the invoice fields from {{content}}.

    2Added: When the total is handwritten,

    3Added: prefer it over the printed one.

    4List every line item.

  5. 05

    Promote

    Promote the tested version to production.

    Promotion copies the content of v6 onto the production template as a new version, v12, and can make it active straight away. The production template keeps its GUID, so anything already calling it picks up the new version without a code change. Development and production can live in separate teams.

    v6 promoted to Production / Invoice Data Extractor as v12 · GUID unchanged

  6. 06

    Live

    Call it from your workflow.

    Your ERP, inbox automation or back-office script posts a document to the template’s trigger URL. BrightEngine answers 202 straight away, runs the active version, and posts the result to your callback URL.

    Add a client request ID and the callback arrives in a signed envelope. Verify it with your team’s webhook secret and you know the result came from BrightEngine and was not replayed.

    Trigger URL

    Use this URL to run this template from external sources.

    https://app.brightengine.example/api/trigger/3f9b2c1e-8a47-4d0e-b6a2-91c0e5d7fd41

    Your system → BrightEngine

    curl -X POST \
      $BRIGHTENGINE_URL/api/trigger/3f9b2c1e-8a47-4d0e-b6a2-91c0e5d7fd41 \
      -H "Content-Type: application/json" \
      -d '{
        "documents": [{
          "url": "https://erp.example.com/inbox/INV-20931.pdf",
          "title": "INV-20931"
        }],
        "data": "Supplier hint: Nordic Supplies AS",
        "callback_url": "https://erp.example.com/hooks/brightengine",
        "client_request_id": "erp-INV-20931"
      }'

    Response202 Accepted

    { "batch_id": 48213, "status": "accepted" }

    BrightEngine → your callback

    X-Brightengine-Timestamp: 1760434262
    X-Brightengine-Signature: sha256=9f2c41…e07b
    
    {
      "schema_version": "1.0",
      "client_request_id": "erp-INV-20931",
      "job_id": "48213",
      "status": "success",
      "result": "{\"invoice_number\":\"INV-20931\",
                 \"total_amount\":12480.0, …}",
      "error": null
    }

    Verify the signature · Node.js

    // HMAC-SHA256 over method, path, timestamp and raw body
    const expected = createHmac("sha256", secret)
      .update(`POST\n${path}\n${timestamp}\n${rawBody}`)
      .digest("hex");
    
    const valid = timingSafeEqual(
      Buffer.from(`sha256=${expected}`),
      Buffer.from(signature),
    );
    // Reject timestamps older than 300 seconds.
  7. 07

    Capture

    Production calls become new cases.

    Every call lands in the API call log with its input, output, status, duration and cost. When an answer is right, capture it as an evaluation case in one click. Capturing says the answer is correct, so BrightEngine never does it on its own unless you tell it to.

    A capture policy picks candidates for you. In review mode it sends matching calls to a review queue. In automatic mode, which you switch on deliberately, it captures them straight away. Try the queue below.

    API Call Logs · Capture Policy · Review queue

  8. 08

    Experiment · live

    Test the next version on real traffic.

    A cheaper model looks promising. A comparison experiment runs it alongside production on real calls, and callers still get the production answer. Run it live for up to 30 days, or against up to 500 past calls.

    Set success criteria before it starts and BrightEngine reports Criteria met, Criteria not met or Inconclusive. With no criteria, you get the evidence without a verdict. Keep any interesting samples as new evaluation cases.

    Comparison experiment · v12 vs v13 · Criteria met

Platform

Everything around the loop.

What the steps above didn’t show: the parts you need once testing before production becomes a habit.

Cost on every run
Input, output and cached tokens with a dollar cost for every run and every judge call, as billed by the provider where available and estimated otherwise.
Teams and roles
Admins, Editors and Members. Keep development and production in separate teams and link them with a promotion target.
Typed questions with System One
Ask Choice, Noul (a probability from 0 to 1) and Score questions through TypeSafe’s Jev models. Each answer arrives with its full probability distribution as model evidence.
Locked templates
Lock a template to prevent unintended changes. Only team admins can unlock it.

REST API · v1

Everything in the app has an endpoint.

Automate the whole loop: create templates and cases, submit evaluations, read results and trigger production runs. Long-running work returns an operation you can poll or receive by callback.

Endpoint What it covers
/api/v1/templatesCreate, version, activate, lock, promote, read API logs
/api/v1/evaluation-casesCreate in batches, bulk update, convert documents to Markdown
/api/v1/evaluationsSubmit, read results and runs, rerun, override a verdict
/api/v1/operations/{id}Poll the status of queued work
/api/v1/teamMembers, invitations, API tokens, provider keys, custom models
/api/trigger/{guid}Run the active version of a template from production
Scoped API tokens
Tokens belong to a team and carry only the abilities you grant. Each token is shown once, when you create it.
templates:read · templates:write · evaluation-cases:write · evaluations:read · evaluations:write · provider-keys:manage · api-tokens:manage
Webhook signing secret
One active secret per team signs every callback with HMAC-SHA256 over the method, path, timestamp and body. Rotate it from team settings.

Start with the examples you already have.

Bring ten real examples and the answers you expect. By the end of the first evaluation you will know where your template stands, field by field.