AI for Document Understanding: Beyond PDF Extraction

Documents are how organizations store and exchange knowledge: contracts, invoices, research papers, technical specifications, forms, reports. The ability to extract structured information and answer questions about documents is one of the most commercially valuable AI capabilities.

The naive approach — extract text, send to LLM — works for simple, well-formatted documents. It fails on complex documents: multi-column layouts, tables with merged cells, scanned forms, documents where visual structure carries meaning, figures with data.

This is the production guide for document understanding that actually works.

What makes document understanding hard

Text extraction alone loses structure. When you extract text from a PDF, you lose:

• Table structure (which cells are headers, which cells are grouped, column relationships)
• Reading order in complex layouts (2-column articles, sidebars, footnotes)
• The relationship between text and figures (“the table in Figure 3 shows…”)
• Handwritten content
• Meaning carried by formatting (bold text as a key term, indentation as hierarchy)

Scanned documents add OCR complexity. Many enterprise documents are scans — PDFs of printed pages, or images of forms. OCR quality varies with scanning quality, font, page orientation, and handwriting presence. OCR errors compound downstream.

Heterogeneous document types. The system that works for dense technical reports may fail on structured forms and vice versa. Production systems often process dozens of document types with different layouts and information patterns.

The evolution of document AI

First generation (OCR + rules): Extract text, apply regular expressions and heuristics to find specific fields. Works for highly structured documents (invoices with consistent templates); fails on variation.

Second generation (layout-aware models): Models like LayoutLM and its successors encode both text content and 2D positional information. A table cell is understood in the context of its position in the grid, not just its text content. Dramatically better on structured forms and tables.

Current generation (vision-language models): Modern MLLMs like Claude, GPT-4V, and Gemini can reason about documents from their visual representation — processing the document as an image and using the visual structure (bounding boxes, visual grouping, font size) to understand organization.

This is the key shift: the document is understood visually, not just textually. The table structure is understood because the model can see it, not because it was separately parsed.

Architecture patterns for production document AI

Pattern 1: Native vision model + direct question answering

Send document pages as images to an MLLM and ask questions:

# Simple but surprisingly powerful
response = client.messages.create(
    model="claude-opus-4-5",
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {"type": "base64", "media_type": "image/jpeg", "data": page_b64}
            },
            {
                "type": "text",
                "text": "Extract all invoice line items as a JSON array with fields: description, quantity, unit_price, total"
            }
        ]
    }]
)

Works well for:

• Single-page or few-page documents
• Documents where visual structure is important (tables, forms)
• Ad hoc question answering on documents
• Scanned documents with irregular layouts

Limitations:

• Context window limits (typically 30-200 pages depending on model/provider)
• Cost scales with page count (image tokens are expensive)
• Not ideal for large document collections requiring retrieval

Pattern 2: Hybrid extraction + LLM pipeline

For production at scale:

1. Structural extraction: Use dedicated document parsing (AWS Textract, Azure Document Intelligence, Unstructured.io) to extract text with layout information — bounding boxes, table structure, reading order.

2. Enrichment: For tables, convert to structured format (markdown tables or JSON). For figures, apply a separate MLLM for caption generation.

3. Chunking with structure awareness: Chunk the document preserving table and section boundaries (don’t split tables across chunks, don’t split section headers from their content).

4. Embedding + retrieval (for collections): Embed chunks and store in a vector database for retrieval.

5. LLM synthesis: Retrieve relevant chunks, provide to LLM with question.

# Structural extraction
from unstructured.partition.pdf import partition_pdf

elements = partition_pdf(
    "document.pdf",
    strategy="hi_res",  # uses vision model for layout
    extract_images_in_pdf=True,
    infer_table_structure=True
)

# Group tables, text, images
tables = [e for e in elements if e.category == "Table"]
text_blocks = [e for e in elements if e.category in ["NarrativeText", "Title"]]

Works well for:

• Large document collections
• Repeated queries over the same documents
• When specific field extraction from consistent document types is needed

Pattern 3: Document-specific fine-tuned models

For high-volume processing of specific document types (invoices, medical records, legal contracts), fine-tuned models that are specialized for your document type and information schema outperform general models on accuracy and cost.

When to use: Processing volume is high enough to justify fine-tuning effort, document type is consistent enough for fine-tuning to generalize, and accuracy requirements are high.

Table extraction

Tables are where most document AI systems fail. The naive approach (extract the text of each cell) loses:

• Row and column headers
• Merged cell structure
• The hierarchical relationship of headers
• Spanning cells

Better approaches:

Vision model native: Ask an MLLM to extract a table directly from an image, outputting it as a markdown table or JSON. Modern models handle this well for typical table layouts.

HTML table extraction: Some document parsers (Camelot for PDFs, Textract for mixed documents) extract tables as structured HTML, preserving cell structure.

Table-specific models: Models fine-tuned on table understanding tasks (TableQA, table-to-text) outperform general models on ambiguous or complex table structures.

Post-processing: Validate extracted tables against expected schema — check that row/column counts are consistent, values are in expected ranges, required fields are present.

Handling specific document challenges

Multi-page tables: Tables that span page breaks are particularly tricky. Most extraction tools treat each page independently. Solutions: stitch extracted tables using header matching, or process adjacent pages together with overlap.

Scanned/handwritten documents: For scanned documents, image quality preprocessing (deskew, denoise, contrast enhancement) before feeding to the model significantly improves results. Handwriting requires models specifically trained on handwritten text (TrOCR, Tesseract with handwriting mode, or cloud services with handwriting support).

Mixed language documents: Documents that mix languages (headers in English, body in French, tables with Spanish labels) require language detection per segment and potentially separate processing.

Redacted documents: Redaction (blacked-out sections) confuses some extraction models. Explicitly handle null/redacted fields in your schema rather than expecting the model to infer content.

Evaluation

Building a solid evaluation set is the most important investment in a document AI system:

1. Representative sample: Include the range of document quality and format variation you’ll encounter in production
2. Ground truth extraction: Manually extract the information your system should produce
3. Metrics per field: Exact match accuracy for discrete fields, fuzzy match for text fields, IoU for extracted regions
4. Failure analysis: Categorize errors (OCR error, layout confusion, missing field, wrong field) to guide improvement

---

Document understanding has advanced dramatically in the 2024-2026 period. The combination of capable vision-language models and specialized document parsing infrastructure has made it feasible to extract rich, structured information from messy real-world documents reliably.

The engineering discipline — combining the right extraction strategy with careful post-processing, validation, and evaluation — determines whether a document AI system works in production or fails on the documents that don’t look like the demos.