- A Datalab account with an API key
- Python 3.11+ installed
- The Datalab SDK:
pip install datalab-python-sdk - Your
DATALAB_API_KEYenvironment variable set
Comparisons are in open beta. The request and response shapes may change. You need
datalab-python-sdk 0.11.0 or later.Run a comparison from Python
This example uploads files, starts a comparison on a pipeline, waits for it to finish, and reads which variant wins:Compare specific variants
Omitvariants to compare the Fast, Balanced, and Accurate modes. To compare
your own settings, give 2 to 4 named variants. A variant may change only the
mode settings:
- For a pipeline of Convert steps:
output_modeandmerge_cross_page. - For a pipeline with one Extract step:
extraction_mode, and the Convert step’soutput_mode.
extraction_mode instead:
collection_id in
place of file_ids. Give exactly one of the two.
Read the result
Call
get_comparison_differences(comparison_id, file_id) for one file to see
each place where the variants disagree: the page, each variant’s reading, the
winning variant(s), and the verdict (decided, none, unclear, or
ignored). For a comparison of Extract, question names the field and
description is the field’s description from your schema. The result also has
fields for a comparison of Extract: every compared field, equal or not, with
each variant’s value in the order of variant_names.
list_comparisons() returns the team’s comparisons, newest first. Filter it with
pipeline_id or collection_id.
CLI
--json to any compare command to print the raw result instead of a summary.
Pricing
A comparison bills $20 per 1,000 pages, once for each document that it compares. Each variant’s own run bills at its own rate on top of that. See pricing.Limits
- 2 to 4 variants per comparison.
- Up to 100 files per comparison: PDFs and images. Spreadsheets have no pages to compare.
- A variant changes only the mode settings described above. For now, a comparison does not check Segment steps or other settings.
Moving from Evals
SDK 0.10.0 called this feature Evals. The old names still work, but they are deprecated and a future version removes them:
The old methods raise a
DeprecationWarning. The old API routes still answer
with eval_id and the evals list key.
Need help?
For questions about comparisons, contact us at support@datalab.to.Next Steps
Compare modes
Start a comparison from the app, and read the results.
Pipelines
Create and run the pipeline that a comparison checks.