Skip to main content
A comparison runs 2 to 4 variants of your pipeline on the same documents. Each variant uses its own settings. The comparison finds the places where their outputs differ, and checks each place against the page. For a pipeline with an Extract step, it compares the extracted values, field by field. The result shows which variant reads your documents best, with its price and speed. Use it to pick the settings for your pipeline. Compare to:
  • Choose between the Fast, Balanced, and Accurate modes for your documents
  • Check whether a setting, such as merging tables across pages, changes the output
  • See exactly where two variants differ before you change a pipeline

Start a comparison

You can start a comparison in three places:
  • From a pipeline: open a pipeline in Forge, upload a document, and click Compare next to Run.
  • From the Compare modes page: open Compare modes, choose a pipeline, and choose the documents or a collection.
  • From the SDK, CLI, or API (open beta): call create_comparison(), run datalab compare run, or send POST /api/v1/pipelines/{pipeline_id}/comparisons. See Compare with the SDK and API.
The Pipelines page lists every comparison that you start under Recent comparisons, however you started it. A pipeline’s Comparisons tab lists the comparisons of that pipeline.

Step 1: Choose what to compare

Add 2 to 4 variants. Each variant runs your pipeline with its own settings. For a pipeline of Convert steps only, a variant sets:
  • a mode: Fast, Balanced, or Accurate
  • Merge across pages, on or off
For a pipeline with a Convert step and one Extract step, a variant sets:
  • the Extract mode: Fast, Balanced, or Accurate
  • the Convert mode, if you want to compare it too
For now, a comparison changes these settings only. Give each variant a different name. Two variants with the same settings are not allowed.

Step 2: Choose the documents

Add up to 100 documents. More documents give a fairer result. Use PDFs or images. Spreadsheets (XLS, XLSX, CSV, ODS) are not compared, because their output has no pages.

Step 3: Run the comparison

Start the comparison. Each variant runs on each document. When all the variants of a document have an output, the comparison checks where they differ. You can close the window. The comparison keeps running, and you can open it from the Pipelines page.

Read the results

The results open with a recommendation, for example “Accurate reads these documents best”. Each variant shows:
  • Quality: “Best”, or the number of places where the variant reads worse than the best variant
  • Price: the variant’s rate per 1,000 pages
  • Speed: seconds per page
  • Tables across page breaks: how many tables that cross a page break the variant keeps whole, when the documents have them
Click Use this on a variant to put its settings in your pipeline.

Review the differences

Click a document to review it. For each page, the review shows the page image and the places where the variants differ, up to 400 per document. For each place, it shows each variant’s reading and what decided it:
  • Decided by the PDF text layer: the PDF has a text layer, and one reading is clearly the closest to it.
  • Decided on the page image: the page has no text layer, or its text layer does not clearly match one reading. A vision model reads the page image and picks the reading that matches what is printed. Table cells and checkboxes are always checked on the page image.
  • No mode matches the page: no variant reads the place as the page does. The review shows it, but it does not count, because it does not tell the variants apart.
  • Unclear: the place could not be decided. It is not counted.

How a comparison checks the output

  • Text: the comparison lines up each variant’s words with the others. Case, whitespace, markdown, and the punctuation around words do not count as differences.
  • Tables: each cell is checked under its row and column headings, so a value in the wrong column counts.
  • Formatting: words that one variant marks bold or italic and another does not.
  • Tables across pages: when you compare merging tables across pages, the comparison checks whether a table that crosses a page break stays one table.
  • Extracted fields: for a pipeline with an Extract step, the comparison checks each field of the schema. Case, whitespace, the format of a number, and an empty value against a missing one do not count as differences. Rows of a list are matched by their values, so a missing row counts once. A vision model reads the pages that the variants cite for the field. It picks the value that the document supports, as the schema’s description asks. When two values are both correct (for example, two formats of one date), both count as correct.
A comparison measures the variants against each other. It does not count a mistake that every variant makes, so the result does not measure absolute quality.

Pricing

A comparison bills $20 per 1,000 pages, once for each document that it compares. Each variant’s run also bills at its own rate. See pricing.

Limitations

  • A comparison checks Convert settings, and Extract settings for a pipeline with one Extract step. Segment comes later.
  • For Extract, a field that no variant cites in a document of more than 4 pages stays unclear. A field description that two careful readers could read in two ways makes the result less certain.
  • Every variant must read the same pages of a document, or the document is not compared.
  • A comparison sends at most 40 places per page to be checked on the page image. Places past that limit stay unclear and are not counted. The review lists at most 400 places per document.
  • Files from an S3 collection source are not compared yet.
  • A scanned page with an OCR text layer is checked against that text layer, including its OCR errors.

Need help?

For questions about comparisons, contact us at support@datalab.to.

Next Steps

Compare with the SDK and API

Start a comparison from Python or the CLI, and read the result in code (open beta).

Pipelines

Create and run the pipeline that a comparison checks.