Extraction Confidence Scoring

Score your structured extraction results to get per-field confidence ratings (1–5) with reasoning that explains what evidence was found or missing.

Extraction scoring is in beta.We’d love your feedback — reach out at support@datalab.to.Scoring is free.

Before you begin, make sure you have:

A Datalab account with an API key (new accounts include $5 in free credits)
Python 3.10+ installed
The Datalab SDK: pip install datalab-python-sdk
Your DATALAB_API_KEY environment variable set

EU requests return a signed result_url; the REST examples below download it before reading the output. See Regional Result Downloads, including the Python SDK 0.5.0 limitation.

How It Works

Confidence scoring runs in cloud fast extraction mode only. When extraction_mode is omitted, the team’s configured default applies. To receive scores you must request fast mode explicitly: extraction_mode="fast". In turbo, balanced, and accurate modes, no numeric scores are produced; extraction_score_average can be absent or null (see the note below).

When you run extraction with extraction_mode="fast", scoring runs automatically afterward. When you poll request_check_url, the extraction result initially contains just the extracted fields and citations. After scoring succeeds, later polls include _score fields and a non-null extraction_score_average. Scoring can fail independently of extraction, so scores are not guaranteed to arrive. A missing or null average means scores are unavailable, not zero confidence. Each scored field receives:

A score from 1 (very low confidence) to 5 (high confidence)
A reasoning string explaining what evidence supports or undermines the extracted value

Beyond setting extraction_mode="fast", no extra parameters or endpoints are needed. Use a bounded polling window and handle unavailable scores separately from low scores.

Using balanced or accurate extraction mode? These modes do not produce numeric confidence scores. Instead they include per-field verification (_meta.verification, with a status of PASS/FAIL_* and feedback) that runs inline as part of the extraction pipeline — a richer, different signal than the numeric confidence scores described here. The two mechanisms are mutually exclusive: use fast mode for numeric _scores, or balanced/accurate mode for inline verification. See Balanced and Accurate Modes.

Example

import requests, json, time, os

headers = {"X-API-Key": os.getenv("DATALAB_API_KEY")}

schema = {
    "type": "object",
    "properties": {
        "invoice_number": {"type": "string", "description": "Invoice ID or number"},
        "total_amount": {"type": "number", "description": "Total amount due"},
        "vendor_name": {"type": "string", "description": "Vendor or company name"}
    },
    "required": ["invoice_number", "total_amount"]
}

with open("invoice.pdf", "rb") as f:
    resp = requests.post(
        "https://www.datalab.to/api/v1/extract",
        files={"file": ("invoice.pdf", f, "application/pdf")},
        data={
            "page_schema": json.dumps(schema),
            "extraction_mode": "fast"  # scoring runs in fast mode only
        },
        headers=headers
    )
check_url = resp.json()["request_check_url"]

# Poll until extraction is complete
while True:
    result = requests.get(check_url, headers=headers).json()
    if result.get("success") is False or result.get("status") == "failed":
        raise RuntimeError(result.get("error") or "Document processing failed")
    if result["status"] == "complete":
        if result.get("result_url"):
            download = requests.get(result["result_url"], timeout=60)
            download.raise_for_status()
            result = {**download.json(), **{k: v for k, v in result.items() if v is not None}}
        if result.get("success") is False:
            raise RuntimeError(result.get("error") or "Document processing failed")
        extracted = result["extraction_schema_json"]
        if isinstance(extracted, str):
            extracted = json.loads(extracted)
        print("Extraction:", extracted)
        break
    time.sleep(2)

# Scores are enriched asynchronously after completion. Keep polling the same
# URL until extraction_score_average is non-null (with a bounded wait).
for _ in range(30):
    if result.get("extraction_score_average") is not None:
        break
    time.sleep(2)
    result = requests.get(check_url, headers=headers).json()

if result.get("extraction_score_average") is None:
    raise TimeoutError("Confidence scores are unavailable; extraction itself succeeded")

if result.get("result_url"):
    download = requests.get(result["result_url"], timeout=60)
    download.raise_for_status()
    result = {**download.json(), **{k: v for k, v in result.items() if v is not None}}

scored = result["extraction_schema_json"]
if isinstance(scored, str):
    scored = json.loads(scored)
for key, value in scored.items():
    if key.endswith("_score"):
        field = key.replace("_score", "")
        print(f"{field}: score={value['score']}, reasoning={value['reasoning']}")

curl -X POST https://www.datalab.to/api/v1/extract \
  -H "X-API-Key: $DATALAB_API_KEY" \
  -F "file=@invoice.pdf" \
  -F 'page_schema={"type":"object","properties":{"invoice_number":{"type":"string","description":"Invoice ID"},"total_amount":{"type":"number","description":"Total due"},"vendor_name":{"type":"string","description":"Vendor or company name"}}}' \
  -F "extraction_mode=fast"

# Poll request_check_url until status is "complete" for extraction results.
# Keep polling the same URL — scores (_score fields + extraction_score_average)
# appear after scoring succeeds. Treat a null average as unavailable and stop
# after your chosen polling deadline. Scoring runs in cloud fast mode only.

Response Format

Without scoring, extraction_schema_json contains fields and citations:

{
  "invoice_number": "INV-2024-001",
  "invoice_number_citations": ["block_123"],
  "total_amount": 1500.00,
  "total_amount_citations": ["block_456"]
}

With scoring, each field also gets a _score object, and the top-level response includes an extraction_score_average:

{
  "invoice_number": "INV-2024-001",
  "invoice_number_citations": ["block_123"],
  "invoice_number_score": {
    "score": 5,
    "reasoning": "Value found verbatim in the document header with a matching citation."
  },
  "total_amount": 1500.00,
  "total_amount_citations": ["block_456"],
  "total_amount_score": {
    "score": 4,
    "reasoning": "Amount found in the totals row; minor ambiguity due to a subtotal nearby."
  }
}

The top-level response also includes extraction_score_average (4.5 in this case), averaging all field scores.

Score Rubric

Score	Meaning
5	High confidence — clear match with strong citation support
4	Good confidence — match found with minor ambiguity
3	Moderate confidence — partial match or uncertain citation
2	Low confidence — match is inferred or weakly supported
1	Very low confidence — no clear evidence found

Using Scores in Practice

Use extraction_score_average for a quick quality check, then inspect individual _score fields to flag low-confidence results:

import json

# After getting scored result (from either approach)
avg = result.get("extraction_score_average")
if avg is None:
    raise ValueError("Confidence scores are unavailable")
print(f"Average score: {avg}")

scored = result["extraction_schema_json"]
if isinstance(scored, str):
    scored = json.loads(scored)
for key, value in scored.items():
    if not key.endswith("_score"):
        continue

    field = key.replace("_score", "")
    if value["score"] <= 2:
        print(f"Low confidence for '{field}': {value['reasoning']}")
    elif value["score"] >= 4:
        print(f"'{field}' = {scored[field]}")

This is useful for building review workflows — auto-accept high-confidence fields and route low-confidence ones to a human reviewer.

Next Steps

Structured Extraction

Full extraction API reference and schema examples

Handling Long Documents

Strategies for extracting from 100+ page documents

Pipelines

Chain processors into versioned, reusable pipelines.

Document Conversion

Convert documents to various formats

General

Document Conversion

Structured Extraction

Document Segmentation

Form Filling

File Management

Pipelines

Create Document

Track Changes

Forge Evals

How It Works

Example

Response Format

Score Rubric

Using Scores in Practice

Next Steps

Structured Extraction

Handling Long Documents

Pipelines

Document Conversion

​How It Works

​Example

​Response Format

​Score Rubric

​Using Scores in Practice

​Next Steps

Structured Extraction

Handling Long Documents

Pipelines

Document Conversion

How It Works

Example

Response Format

Score Rubric

Using Scores in Practice

Next Steps