Evaluate Document Parsers for Complex Documents | Extend

Evaluate Document Parsers on Complex Documents

Kushal Byatnal

5 min read

Apr 13, 2026

Blog Post

Complex production documents fail in relationships, not only characters. Structure-aware document parsing preserves relationships between headings, tables, fields, values, and pages. Developers should evaluate the returned data structure and its evidence. Do not select a system from a generic vendor list or a single accuracy number. Use the AI-agent parser evaluation guide for product selection. Use this guide to test whether a chosen system can preserve the relationships your application needs.

How to evaluate structure-aware document parsing

Use a real document with known values and difficult structure. This example uses the public multi-page bank statement in Extend's API quickstart. The same file is available in the site's public document bank, with a saved Parse 2.0 output for inspection.

Evaluate these signals independently:

Reading order: Headings, account summaries, and transaction tables must appear in the order a person reads them.
Nested objects and arrays: Each account must own its balances and transactions.
Parent-child relationships: Every transaction must reference the correct parent account.
Repeated records: The transaction count must match the ground truth across all pages.
Cross-page references: A continued table must retain the correct header and account context.
Source citations: Each material field must point to the correct page and region.
Field confidence: Confidence or review signals must be available at the field path.
Schema validation: The final value must conform to the declared schema and business rules.

Failure modes and required verification

Failure mode	Required output signal	Verification method
Column or section order changes	Ordered chunks or blocks with page metadata	Compare the parsed sequence with human reading order
A child field is attached to the wrong parent	Nested object or explicit stable parent key	Assert that every child key matches its enclosing object
Repeated rows are omitted or duplicated	Complete array with stable cardinality	Compare array length and row identifiers with ground truth
A table continues on another page	Page-aware rows with retained table and parent context	Test records before and after each page boundary
A plausible value comes from the wrong region	Field-level citation with page and bounding polygon	Render the citation over the source PDF and inspect the region
A low-quality value passes automatically	Field-level review or confidence signal	Apply a fixed threshold and measure review precision and recall
The response shape drifts	Versioned JSON Schema and schema-valid output	Validate every response before the next workflow step
Totals do not reconcile	Explicit business-rule result	Recalculate totals from the returned rows and compare balances

Complete schema and API call

The schema below represents one statement with a parent account and repeated transactions. Each transaction repeats the parent account number. That deliberate redundancy makes a wrong parent-child link testable.

from extend_ai import Extend

client = Extend()

schema = {
    "type": "object",
    "properties": {
        "statement_period": {
            "type": "object",
            "properties": {
                "start_date": {"type": ["string", "null"], "extend:type": "date"},
                "end_date": {"type": ["string", "null"], "extend:type": "date"},
            },
            "required": ["start_date", "end_date"],
        },
        "account": {
            "type": "object",
            "properties": {
                "account_number": {"type": ["string", "null"]},
                "beginning_balance": {"type": ["number", "null"]},
                "ending_balance": {"type": ["number", "null"]},
                "transactions": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "posted_date": {
                                "type": ["string", "null"],
                                "extend:type": "date",
                            },
                            "description": {"type": ["string", "null"]},
                            "amount": {"type": ["number", "null"]},
                            "running_balance": {"type": ["number", "null"]},
                            "parent_account_number": {
                                "type": ["string", "null"],
                                "description": "The account number that owns this transaction",
                            },
                        },
                        "required": ["posted_date", "description", "amount", "running_balance", "parent_account_number"],
                    },
                },
            },
            "required": ["account_number", "beginning_balance", "ending_balance", "transactions"],
        },
    },
    "required": ["statement_period", "account"],
}

run = client.extract(
    file={
        "url": "https://extend-public-files.s3.us-east-2.amazonaws.com/bank_statement_example.pdf"
    },
    config={
        "schema": schema,
        "advancedOptions": {"citationsEnabled": True},
    },
)

The schema documentation defines supported nested objects, arrays, nullable fields, and typed dates. The extraction overview explains how Extract uses parsed content. The synchronous call is suitable for testing. Use asynchronous runs and webhooks for large files and production volume.

Abbreviated structure-aware response

The API returns values and field metadata under matching paths. This abbreviated shape uses values visible in the public sample. Exact confidence values and polygons vary by run.

{
  "value": {
    "statement_period": {
      "start_date": "2020-12-22",
      "end_date": "2021-01-25"
    },
    "account": {
      "account_number": "000000861006133",
      "beginning_balance": 438.59,
      "ending_balance": 459.25,
      "transactions": [
        {
          "posted_date": "2020-12-24",
          "description": "Discover E-Payment 6930",
          "amount": -27.90,
          "running_balance": 410.69,
          "parent_account_number": "000000861006133"
        },
        {
          "posted_date": "2020-12-29",
          "description": "ATM Cash Deposit",
          "amount": 101.00,
          "running_balance": 438.60,
          "parent_account_number": "000000861006133"
        }
      ]
    }
  },
  "metadata": {
    "account.account_number": {
      "citations": [{"page": {"number": 1}, "referenceText": "000000861006133"}]
    },
    "account.transactions[0].amount": {
      "citations": [{"page": {"number": 1}, "referenceText": "-27.90"}]
    }
  }
}

The nested object proves ownership. The repeated parent_account_number makes each row independently testable. The metadata path binds a field to its source. See the current response format for the complete citation, confidence, and review shape.

Validation rule before the next agent action

Do not send extracted data to an agent because the response is valid JSON. Validate the relationships and the business rule first.

from decimal import Decimal

value = run.output.value
metadata = run.output.metadata
account = value["account"]

assert all(
    row["parent_account_number"] == account["account_number"]
    for row in account["transactions"]
)
assert all(
    metadata.get(f"account.transactions[{index}].amount", {}).get("citations")
    for index in range(len(account["transactions"]))
)

net_change = sum(
    Decimal(str(row["amount"])) for row in account["transactions"]
)
assert Decimal(str(account["beginning_balance"])) + net_change == Decimal(
    str(account["ending_balance"])
)

next_agent_action = "post_statement"  # Run only after all assertions pass.

In a production workflow, a failed assertion should route the document to review. The workflow must not silently discard a repeated row or accept a value from the wrong page.

Which benchmark answers which question

RealDoc-Bench measures parsing layout with Adjusted F1 and document Q&A with field-level accuracy. The public corpus includes 1,500 layout samples and 1,359 prompts across 581 documents in financial services, real estate, logistics, and healthcare. It does not measure business-schema extraction or every private document type.

LongArray-Extract measures complete repeated-record extraction on 45 financial, clinical, and legal PDFs. Failed and timed-out runs score zero. It tests array cardinality and field accuracy, not generic RAG retrieval or PDF-to-Markdown quality.

Use the benchmark hub for methodology and public data. For product selection, return to the AI-agent parser evaluation guide. For implementation, use the extraction docs, evaluation docs, and workflow docs.

A private-corpus test plan

Define the exact schema, required relationships, and business rules.
Select documents with page breaks, repeated records, nested tables, and known scan defects.
Label every expected value and parent-child link before the test.
Run the same product mode and configuration across the full corpus.
Count missing rows, duplicates, wrong parents, failures, and timeouts.
Render citations for every material error.
Measure review volume at the threshold your workflow will use.
Ship only after the schema and business-rule checks pass on held-out documents.

FAQ

What is structure-aware document parsing?

Structure-aware document parsing preserves relationships between headings, tables, fields, values, and pages. Structured extraction can then keep each value attached to its parent object, repeated record, and source region.

How do I test nested and repeated fields?

Define the target schema before the run. Score parent-child links, array cardinality, field values, source citations, and schema validity separately.