> ## Documentation Index
> Fetch the complete documentation index at: https://documentation.datalab.to/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals with the SDK and API

> Run an Eval from code: compare pipeline variants, read which one wins, and see why

**Before you begin**, make sure you have:

1. A [Datalab account](https://www.datalab.to/auth/sign_up) with an [API key](https://www.datalab.to/app/keys)
2. Python 3.11+ installed
3. The Datalab SDK: `pip install datalab-python-sdk`
4. Your `DATALAB_API_KEY` environment variable set

> [!NOTE]
>
> Evals are in open beta. The request and response shapes may change.

An Eval runs 2 to 4 variants of a pipeline on the same files, then compares their
outputs. For a pipeline of Convert steps, it compares the pages. For a pipeline
with one Extract step, it compares the extracted fields. Each place where the
variants differ gets a verdict: which variant reads it as the page does.

## Run an Eval from Python

This example uploads files, starts an Eval on a pipeline, waits for it to
finish, and reads which variant wins:

```python Python SDK theme={null}
from datalab_sdk import DatalabClient

client = DatalabClient()

files = client.upload_files(["invoice_1.pdf", "invoice_2.pdf"])
file_ids = [f.file_id for f in files]

evaluation = client.create_eval("pl_abc123", file_ids=file_ids)
evaluation = client.wait_for_eval(evaluation.eval_id)

best = max(evaluation.variants, key=lambda v: v.share or 0)
print(f"{best.name} wins: share={best.share}, "
      f"rate={best.rate_per_1000_pages_cents}c/1000 pages, "
      f"speed={best.seconds_per_page}s/page")

# Where the winning variant differs from the others on one file
diffs = client.get_eval_differences(evaluation.eval_id, file_ids[0])
for d in diffs.differences:
    print(d.page, d.verdict, d.question or d.before[:40])
```

```bash cURL theme={null}
curl -X POST https://www.datalab.to/api/v1/pipelines/pl_abc123/evals \
  -H "X-API-Key: $DATALAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"file_ids": [1, 2]}'

# Poll until status is "completed" or "failed"
curl https://www.datalab.to/api/v1/evals/EVAL_ID \
  -H "X-API-Key: $DATALAB_API_KEY"

curl https://www.datalab.to/api/v1/evals/EVAL_ID/files/FILE_ID/differences \
  -H "X-API-Key: $DATALAB_API_KEY"
```

```python Python (requests) theme={null}
import os, time, requests

BASE = "https://www.datalab.to/api/v1"
headers = {"X-API-Key": os.getenv("DATALAB_API_KEY")}

resp = requests.post(
    f"{BASE}/pipelines/pl_abc123/evals",
    headers=headers,
    json={"file_ids": [1, 2]},
)
eval_id = resp.json()["eval_id"]

while True:
    evaluation = requests.get(f"{BASE}/evals/{eval_id}", headers=headers).json()
    if evaluation["status"] in ("completed", "failed"):
        break
    time.sleep(10)

best = max(evaluation["variants"], key=lambda v: v["share"] or 0)
print(best["name"], best["share"])
```

## Compare specific variants

Omit `variants` to compare the Fast, Balanced, and Accurate modes. To compare
your own settings, give 2 to 4 named variants. A variant may change only the
mode settings:

* For a pipeline of Convert steps: `output_mode` and `merge_cross_page`.
* For a pipeline with one Extract step: `extraction_mode`, and the Convert
  step's `output_mode`.

```python theme={null}
from datalab_sdk import DatalabClient, EvalVariant

client = DatalabClient()

evaluation = client.create_eval(
    "pl_abc123",
    file_ids=file_ids,
    variants=[
        EvalVariant(name="No merge", steps={"convert": {"output_mode": "accurate"}}),
        EvalVariant(
            name="Merge across pages",
            steps={"convert": {"output_mode": "accurate", "merge_cross_page": True}},
        ),
    ],
)
```

For a pipeline with an Extract step, a variant sets `extraction_mode` instead:

```python theme={null}
evaluation = client.create_eval(
    "pl_extract123",
    file_ids=file_ids,
    variants=[
        EvalVariant(name="Fast", steps={"extract": {"extraction_mode": "fast"}}),
        EvalVariant(name="Balanced", steps={"extract": {"extraction_mode": "balanced"}}),
        EvalVariant(name="Accurate", steps={"extract": {"extraction_mode": "accurate"}}),
    ],
)
```

Compare a collection instead of uploaded files by passing `collection_id`
instead of `file_ids`. Give exactly one of the two.

## Read the result

| Field on `EvalVariantResult` | What it means |
| - | - |
| `share` | This variant's share of the decided places (`won` / `decided`). Relative to the other variants, not an absolute score. |
| `rate_per_1000_pages_cents` | This variant's list price per 1,000 pages |
| `seconds_per_page` | This variant's processing speed |
| `page_breaks_passed` / `page_breaks_total` | Tables that cross a page break and stay one table, when you compare Merge across pages |
| `convert_settings` / `extract_settings` | The exact settings this variant ran with |

Call `get_eval_differences(eval_id, file_id)` for one file to see each place
the variants disagree: the page, each variant's reading, the winning
variant(s), and the `verdict` (`decided`, `none`, `unclear`, or `ignored`).
For an Eval of Extract, `question` names the field and `description` is the
field's description from your schema.
`get_eval_differences()` also returns `fields` for an Eval of Extract: every
compared field, equal or not, with each variant's value in the order of
`variant_names`.

## CLI

```bash Start and wait theme={null}
datalab eval run invoice_1.pdf invoice_2.pdf --pipeline pl_abc123 --wait
```

```bash Compare specific variants theme={null}
datalab eval run invoice_1.pdf --pipeline pl_abc123 \
  --variant-json '[{"name":"Fast","steps":{"extract":{"extraction_mode":"fast"}}},
                   {"name":"Balanced","steps":{"extract":{"extraction_mode":"balanced"}}}]' \
  --wait
```

```bash Check status and differences theme={null}
datalab eval get EVAL_ID
datalab eval list --pipeline_id pl_abc123
datalab eval differences EVAL_ID FILE_ID
```

Add `--json` to any `eval` command to print the raw result instead of a summary.

## Pricing

The comparison bills \$20 per 1,000 pages, once for each document that it
compares. Each variant's own run bills at its own rate on top of that. See
[pricing](https://www.datalab.to/pricing).

## Limits

* 2 to 4 variants per Eval.
* Up to 100 files per Eval: PDFs and images. Spreadsheets have no pages to compare.
* A variant changes only the mode settings described above. For now, an Eval
  does not compare Segment steps or other settings.

## Need help?

For questions about Evals or help with a custom evaluation, contact us at
[support@datalab.to](mailto:support@datalab.to).

## Next Steps

> [card] [Forge Evals overview](https://documentation.datalab.to/docs/recipes/forge-evals/overview.md)
>
> Start an Eval from the app, and how to read the results.

> [card] [Pipelines](https://documentation.datalab.to/docs/welcome/sdk/pipelines.md)
>
> Create and run the pipeline that an Eval compares.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.