> ## Documentation Index
> Fetch the complete documentation index at: https://documentation.datalab.to/llms.txt
> Use this file to discover all available pages before exploring further.

# Compare with the SDK and API

> Start a comparison from code: run pipeline variants, read which one wins, and see why

**Before you begin**, make sure you have:

1. A [Datalab account](https://www.datalab.to/auth/sign_up) with an [API key](https://www.datalab.to/app/keys)
2. Python 3.11+ installed
3. The Datalab SDK: `pip install datalab-python-sdk`
4. Your `DATALAB_API_KEY` environment variable set

> [!NOTE]
>
> Comparisons are in open beta. The request and response shapes may change. You need `datalab-python-sdk` 0.11.0 or later.

A comparison runs 2 to 4 variants of a pipeline on the same files, then checks
their outputs. For a pipeline of Convert steps, it compares the pages. For a
pipeline with one Extract step, it compares the extracted fields. Each place
where the variants differ gets a verdict: which variant reads it as the page does.

## Run a comparison from Python

This example uploads files, starts a comparison on a pipeline, waits for it to
finish, and reads which variant wins:

```python Python SDK theme={null}
from datalab_sdk import DatalabClient

client = DatalabClient()

files = client.upload_files(["invoice_1.pdf", "invoice_2.pdf"])
file_ids = [f.file_id for f in files]

comparison = client.create_comparison("pl_abc123", file_ids=file_ids)
comparison = client.wait_for_comparison(comparison.comparison_id)

best = max(comparison.variants, key=lambda v: v.share or 0)
print(f"{best.name} wins: share={best.share}, "
      f"rate={best.rate_per_1000_pages_cents}c/1000 pages, "
      f"speed={best.seconds_per_page}s/page")

# Where the variants differ on one file
diffs = client.get_comparison_differences(comparison.comparison_id, file_ids[0])
for d in diffs.differences:
    print(d.page, d.verdict, d.question or d.before[:40])
```

```bash cURL theme={null}
curl -X POST https://www.datalab.to/api/v1/pipelines/pl_abc123/comparisons \
  -H "X-API-Key: $DATALAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"file_ids": [1, 2]}'

# Poll until status is "completed" or "failed"
curl https://www.datalab.to/api/v1/comparisons/COMPARISON_ID \
  -H "X-API-Key: $DATALAB_API_KEY"

curl https://www.datalab.to/api/v1/comparisons/COMPARISON_ID/files/FILE_ID/differences \
  -H "X-API-Key: $DATALAB_API_KEY"
```

```python Python (requests) theme={null}
import os, time, requests

BASE = "https://www.datalab.to/api/v1"
headers = {"X-API-Key": os.getenv("DATALAB_API_KEY")}

resp = requests.post(
    f"{BASE}/pipelines/pl_abc123/comparisons",
    headers=headers,
    json={"file_ids": [1, 2]},
)
comparison_id = resp.json()["comparison_id"]

while True:
    comparison = requests.get(f"{BASE}/comparisons/{comparison_id}", headers=headers).json()
    if comparison["status"] in ("completed", "failed"):
        break
    time.sleep(10)

best = max(comparison["variants"], key=lambda v: v["share"] or 0)
print(best["name"], best["share"])
```

## Compare specific variants

Omit `variants` to compare the Fast, Balanced, and Accurate modes. To compare
your own settings, give 2 to 4 named variants. A variant may change only the
mode settings:

* For a pipeline of Convert steps: `output_mode` and `merge_cross_page`.
* For a pipeline with one Extract step: `extraction_mode`, and the Convert
  step's `output_mode`.

```python theme={null}
from datalab_sdk import ComparisonVariant, DatalabClient

client = DatalabClient()

comparison = client.create_comparison(
    "pl_abc123",
    file_ids=file_ids,
    variants=[
        ComparisonVariant(name="No merge", steps={"convert": {"output_mode": "accurate"}}),
        ComparisonVariant(
            name="Merge across pages",
            steps={"convert": {"output_mode": "accurate", "merge_cross_page": True}},
        ),
    ],
)
```

For a pipeline with an Extract step, a variant sets `extraction_mode` instead:

```python theme={null}
comparison = client.create_comparison(
    "pl_extract123",
    file_ids=file_ids,
    variants=[
        ComparisonVariant(name="Fast", steps={"extract": {"extraction_mode": "fast"}}),
        ComparisonVariant(name="Balanced", steps={"extract": {"extraction_mode": "balanced"}}),
        ComparisonVariant(name="Accurate", steps={"extract": {"extraction_mode": "accurate"}}),
    ],
)
```

To compare a collection instead of uploaded files, pass `collection_id` in
place of `file_ids`. Give exactly one of the two.

## Read the result

| Field on `ComparisonVariantResult` | What it means |
| - | - |
| `share` | This variant's share of the decided places (`won` / `decided`). Relative to the other variants, not an absolute score. |
| `rate_per_1000_pages_cents` | This variant's list price per 1,000 pages |
| `seconds_per_page` | This variant's processing speed |
| `page_breaks_passed` / `page_breaks_total` | Tables that cross a page break and stay one table, when you compare Merge across pages |
| `convert_settings` / `extract_settings` | The exact settings this variant ran with |

Call `get_comparison_differences(comparison_id, file_id)` for one file to see
each place where the variants disagree: the page, each variant's reading, the
winning variant(s), and the `verdict` (`decided`, `none`, `unclear`, or
`ignored`). For a comparison of Extract, `question` names the field and
`description` is the field's description from your schema. The result also has
`fields` for a comparison of Extract: every compared field, equal or not, with
each variant's value in the order of `variant_names`.

`list_comparisons()` returns the team's comparisons, newest first. Filter it with
`pipeline_id` or `collection_id`.

## CLI

```bash Start and wait theme={null}
datalab compare run invoice_1.pdf invoice_2.pdf --pipeline pl_abc123 --wait
```

```bash Compare specific variants theme={null}
datalab compare run invoice_1.pdf --pipeline pl_abc123 \
  --variant-json '[{"name":"Fast","steps":{"extract":{"extraction_mode":"fast"}}},
                   {"name":"Balanced","steps":{"extract":{"extraction_mode":"balanced"}}}]' \
  --wait
```

```bash Check status and differences theme={null}
datalab compare get COMPARISON_ID
datalab compare list --pipeline_id pl_abc123
datalab compare differences COMPARISON_ID FILE_ID
```

Add `--json` to any `compare` command to print the raw result instead of a summary.

## Pricing

A comparison bills \$20 per 1,000 pages, once for each document that it
compares. Each variant's own run bills at its own rate on top of that. See
[pricing](https://www.datalab.to/pricing).

## Limits

* 2 to 4 variants per comparison.
* Up to 100 files per comparison: PDFs and images. Spreadsheets have no pages to compare.
* A variant changes only the mode settings described above. For now, a
  comparison does not check Segment steps or other settings.

## Moving from Evals

SDK 0.10.0 called this feature Evals. The old names still work, but they are
deprecated and a future version removes them:

| Old name | New name |
| - | - |
| `create_eval()`, `get_eval()`, `list_evals()` | `create_comparison()`, `get_comparison()`, `list_comparisons()` |
| `get_eval_differences()`, `wait_for_eval()` | `get_comparison_differences()`, `wait_for_comparison()` |
| `Eval`, `EvalVariant`, and the other `Eval*` models | `Comparison`, `ComparisonVariant`, and the other `Comparison*` models |
| `eval_id`, `EvalList.evals` | `comparison_id`, `ComparisonList.comparisons` |
| `datalab eval ...` | `datalab compare ...` |
| `/api/v1/pipelines/{pipeline_id}/evals`, `/api/v1/evals/...` | `/api/v1/pipelines/{pipeline_id}/comparisons`, `/api/v1/comparisons/...` |

The old methods raise a `DeprecationWarning`. The old API routes still answer
with `eval_id` and the `evals` list key.

## Need help?

For questions about comparisons, contact us at
[support@datalab.to](mailto:support@datalab.to).

## Next Steps

> [card] [Compare modes](https://documentation.datalab.to/docs/recipes/compare/overview.md)
>
> Start a comparison from the app, and read the results.

> [card] [Pipelines](https://documentation.datalab.to/docs/welcome/sdk/pipelines.md)
>
> Create and run the pipeline that a comparison checks.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.