Evals with the SDK and API

Before you begin, make sure you have:

A Datalab account with an API key
Python 3.11+ installed
The Datalab SDK: pip install datalab-python-sdk
Your DATALAB_API_KEY environment variable set

Evals are in open beta. The request and response shapes may change.

An Eval runs 2 to 4 variants of a pipeline on the same files, then compares their outputs. For a pipeline of Convert steps, it compares the pages. For a pipeline with one Extract step, it compares the extracted fields. Each place where the variants differ gets a verdict: which variant reads it as the page does.

Run an Eval from Python

This example uploads files, starts an Eval on a pipeline, waits for it to finish, and reads which variant wins:

from datalab_sdk import DatalabClient

client = DatalabClient()

files = client.upload_files(["invoice_1.pdf", "invoice_2.pdf"])
file_ids = [f.file_id for f in files]

evaluation = client.create_eval("pl_abc123", file_ids=file_ids)
evaluation = client.wait_for_eval(evaluation.eval_id)

best = max(evaluation.variants, key=lambda v: v.share or 0)
print(f"{best.name} wins: share={best.share}, "
      f"rate={best.rate_per_1000_pages_cents}c/1000 pages, "
      f"speed={best.seconds_per_page}s/page")

# Where the winning variant differs from the others on one file
diffs = client.get_eval_differences(evaluation.eval_id, file_ids[0])
for d in diffs.differences:
    print(d.page, d.verdict, d.question or d.before[:40])

curl -X POST https://www.datalab.to/api/v1/pipelines/pl_abc123/evals \
  -H "X-API-Key: $DATALAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"file_ids": [1, 2]}'

# Poll until status is "completed" or "failed"
curl https://www.datalab.to/api/v1/evals/EVAL_ID \
  -H "X-API-Key: $DATALAB_API_KEY"

curl https://www.datalab.to/api/v1/evals/EVAL_ID/files/FILE_ID/differences \
  -H "X-API-Key: $DATALAB_API_KEY"

import os, time, requests

BASE = "https://www.datalab.to/api/v1"
headers = {"X-API-Key": os.getenv("DATALAB_API_KEY")}

resp = requests.post(
    f"{BASE}/pipelines/pl_abc123/evals",
    headers=headers,
    json={"file_ids": [1, 2]},
)
eval_id = resp.json()["eval_id"]

while True:
    evaluation = requests.get(f"{BASE}/evals/{eval_id}", headers=headers).json()
    if evaluation["status"] in ("completed", "failed"):
        break
    time.sleep(10)

best = max(evaluation["variants"], key=lambda v: v["share"] or 0)
print(best["name"], best["share"])

Compare specific variants

Omit variants to compare the Fast, Balanced, and Accurate modes. To compare your own settings, give 2 to 4 named variants. A variant may change only the mode settings:

For a pipeline of Convert steps: output_mode and merge_cross_page.
For a pipeline with one Extract step: extraction_mode, and the Convert step’s output_mode.

from datalab_sdk import DatalabClient, EvalVariant

client = DatalabClient()

evaluation = client.create_eval(
    "pl_abc123",
    file_ids=file_ids,
    variants=[
        EvalVariant(name="No merge", steps={"convert": {"output_mode": "accurate"}}),
        EvalVariant(
            name="Merge across pages",
            steps={"convert": {"output_mode": "accurate", "merge_cross_page": True}},
        ),
    ],
)

For a pipeline with an Extract step, a variant sets extraction_mode instead:

evaluation = client.create_eval(
    "pl_extract123",
    file_ids=file_ids,
    variants=[
        EvalVariant(name="Fast", steps={"extract": {"extraction_mode": "fast"}}),
        EvalVariant(name="Balanced", steps={"extract": {"extraction_mode": "balanced"}}),
        EvalVariant(name="Accurate", steps={"extract": {"extraction_mode": "accurate"}}),
    ],
)

Compare a collection instead of uploaded files by passing collection_id instead of file_ids. Give exactly one of the two.

Read the result

Field on `EvalVariantResult`	What it means
`share`	This variant’s share of the decided places (`won` / `decided`). Relative to the other variants, not an absolute score.
`rate_per_1000_pages_cents`	This variant’s list price per 1,000 pages
`seconds_per_page`	This variant’s processing speed
`page_breaks_passed` / `page_breaks_total`	Tables that cross a page break and stay one table, when you compare Merge across pages
`convert_settings` / `extract_settings`	The exact settings this variant ran with

Call get_eval_differences(eval_id, file_id) for one file to see each place the variants disagree: the page, each variant’s reading, the winning variant(s), and the verdict (decided, none, unclear, or ignored). For an Eval of Extract, question names the field and description is the field’s description from your schema. get_eval_differences() also returns fields for an Eval of Extract: every compared field, equal or not, with each variant’s value in the order of variant_names.

CLI

datalab eval run invoice_1.pdf invoice_2.pdf --pipeline pl_abc123 --wait

datalab eval run invoice_1.pdf --pipeline pl_abc123 \
  --variant-json '[{"name":"Fast","steps":{"extract":{"extraction_mode":"fast"}}},
                   {"name":"Balanced","steps":{"extract":{"extraction_mode":"balanced"}}}]' \
  --wait

datalab eval get EVAL_ID
datalab eval list --pipeline_id pl_abc123
datalab eval differences EVAL_ID FILE_ID

Add --json to any eval command to print the raw result instead of a summary.

Pricing

The comparison bills $20 per 1,000 pages, once for each document that it compares. Each variant’s own run bills at its own rate on top of that. See pricing.

Limits

2 to 4 variants per Eval.
Up to 100 files per Eval: PDFs and images. Spreadsheets have no pages to compare.
A variant changes only the mode settings described above. For now, an Eval does not compare Segment steps or other settings.

Need help?

For questions about Evals or help with a custom evaluation, contact us at support@datalab.to.

General

Document Conversion

Structured Extraction

Document Segmentation

Form Fields

Form Filling

Redaction

Document Agent

File Management

Pipelines

Create Document

Track Changes

Forge Evals

Run an Eval from Python

Compare specific variants

Read the result

CLI

Pricing

Limits

Need help?

Next Steps

Forge Evals overview

Pipelines