- Choose between the Fast, Balanced, and Accurate modes for your documents
- Check whether a setting, such as merging tables across pages, changes the output
- See exactly where two variants differ before you change a pipeline
Start a comparison
You can start a comparison in three places:- From a pipeline: open a pipeline in Forge, upload a document, and click Compare next to Run.
- From the Compare modes page: open Compare modes, choose a pipeline, and choose the documents or a collection.
- From the SDK, CLI, or API (open beta): call
create_comparison(), rundatalab compare run, or sendPOST /api/v1/pipelines/{pipeline_id}/comparisons. See Compare with the SDK and API.
Step 1: Choose what to compare
Add 2 to 4 variants. Each variant runs your pipeline with its own settings. For a pipeline of Convert steps only, a variant sets:- a mode: Fast, Balanced, or Accurate
- Merge across pages, on or off
- the Extract mode: Fast, Balanced, or Accurate
- the Convert mode, if you want to compare it too
Step 2: Choose the documents
Add up to 100 documents. More documents give a fairer result. Use PDFs or images. Spreadsheets (XLS, XLSX, CSV, ODS) are not compared, because their output has no pages.Step 3: Run the comparison
Start the comparison. Each variant runs on each document. When all the variants of a document have an output, the comparison checks where they differ. You can close the window. The comparison keeps running, and you can open it from the Pipelines page.Read the results
The results open with a recommendation, for example “Accurate reads these documents best”. Each variant shows:- Quality: “Best”, or the number of places where the variant reads worse than the best variant
- Price: the variant’s rate per 1,000 pages
- Speed: seconds per page
- Tables across page breaks: how many tables that cross a page break the variant keeps whole, when the documents have them
Review the differences
Click a document to review it. For each page, the review shows the page image and the places where the variants differ, up to 400 per document. For each place, it shows each variant’s reading and what decided it:- Decided by the PDF text layer: the PDF has a text layer, and one reading is clearly the closest to it.
- Decided on the page image: the page has no text layer, or its text layer does not clearly match one reading. A vision model reads the page image and picks the reading that matches what is printed. Table cells and checkboxes are always checked on the page image.
- No mode matches the page: no variant reads the place as the page does. The review shows it, but it does not count, because it does not tell the variants apart.
- Unclear: the place could not be decided. It is not counted.
How a comparison checks the output
- Text: the comparison lines up each variant’s words with the others. Case, whitespace, markdown, and the punctuation around words do not count as differences.
- Tables: each cell is checked under its row and column headings, so a value in the wrong column counts.
- Formatting: words that one variant marks bold or italic and another does not.
- Tables across pages: when you compare merging tables across pages, the comparison checks whether a table that crosses a page break stays one table.
- Extracted fields: for a pipeline with an Extract step, the comparison checks each field of the schema. Case, whitespace, the format of a number, and an empty value against a missing one do not count as differences. Rows of a list are matched by their values, so a missing row counts once. A vision model reads the pages that the variants cite for the field. It picks the value that the document supports, as the schema’s description asks. When two values are both correct (for example, two formats of one date), both count as correct.
A comparison measures the variants against each other. It does not count a mistake that every variant makes, so the result does not measure absolute quality.
Pricing
A comparison bills $20 per 1,000 pages, once for each document that it compares. Each variant’s run also bills at its own rate. See pricing.Limitations
- A comparison checks Convert settings, and Extract settings for a pipeline with one Extract step. Segment comes later.
- For Extract, a field that no variant cites in a document of more than 4 pages stays unclear. A field description that two careful readers could read in two ways makes the result less certain.
- Every variant must read the same pages of a document, or the document is not compared.
- A comparison sends at most 40 places per page to be checked on the page image. Places past that limit stay unclear and are not counted. The review lists at most 400 places per document.
- Files from an S3 collection source are not compared yet.
- A scanned page with an OCR text layer is checked against that text layer, including its OCR errors.
Need help?
For questions about comparisons, contact us at support@datalab.to.Next Steps
Compare with the SDK and API
Start a comparison from Python or the CLI, and read the result in code (open beta).
Pipelines
Create and run the pipeline that a comparison checks.