Skip to main content
Before you begin, make sure you have:
  1. A Datalab account with an API key
  2. Python 3.11+ installed
  3. The Datalab SDK: pip install datalab-python-sdk
  4. Your DATALAB_API_KEY environment variable set
Comparisons are in open beta. The request and response shapes may change. You need datalab-python-sdk 0.11.0 or later.
A comparison runs 2 to 4 variants of a pipeline on the same files, then checks their outputs. For a pipeline of Convert steps, it compares the pages. For a pipeline with one Extract step, it compares the extracted fields. Each place where the variants differ gets a verdict: which variant reads it as the page does.

Run a comparison from Python

This example uploads files, starts a comparison on a pipeline, waits for it to finish, and reads which variant wins:

Compare specific variants

Omit variants to compare the Fast, Balanced, and Accurate modes. To compare your own settings, give 2 to 4 named variants. A variant may change only the mode settings:
  • For a pipeline of Convert steps: output_mode and merge_cross_page.
  • For a pipeline with one Extract step: extraction_mode, and the Convert step’s output_mode.
For a pipeline with an Extract step, a variant sets extraction_mode instead:
To compare a collection instead of uploaded files, pass collection_id in place of file_ids. Give exactly one of the two.

Read the result

Call get_comparison_differences(comparison_id, file_id) for one file to see each place where the variants disagree: the page, each variant’s reading, the winning variant(s), and the verdict (decided, none, unclear, or ignored). For a comparison of Extract, question names the field and description is the field’s description from your schema. The result also has fields for a comparison of Extract: every compared field, equal or not, with each variant’s value in the order of variant_names. list_comparisons() returns the team’s comparisons, newest first. Filter it with pipeline_id or collection_id.

CLI

Add --json to any compare command to print the raw result instead of a summary.

Pricing

A comparison bills $20 per 1,000 pages, once for each document that it compares. Each variant’s own run bills at its own rate on top of that. See pricing.

Limits

  • 2 to 4 variants per comparison.
  • Up to 100 files per comparison: PDFs and images. Spreadsheets have no pages to compare.
  • A variant changes only the mode settings described above. For now, a comparison does not check Segment steps or other settings.

Moving from Evals

SDK 0.10.0 called this feature Evals. The old names still work, but they are deprecated and a future version removes them: The old methods raise a DeprecationWarning. The old API routes still answer with eval_id and the evals list key.

Need help?

For questions about comparisons, contact us at support@datalab.to.

Next Steps

Compare modes

Start a comparison from the app, and read the results.

Pipelines

Create and run the pipeline that a comparison checks.