OCR + Document Processing

Document processing converts unstructured content (scanned PDFs, forms, invoices, contracts) into structured, queryable data. The pattern combines OCR for text extraction, layout models for structure detection, and an LLM for semantic extraction with confidence scoring.

Most demos extract text from one clean PDF and stop. Production systems have to handle degraded scans, multi-page tables, and cross-references between pages - and they have to know when to trust their own output.

The Pattern

Why This Is Hard

Three failure modes are invisible in demos and unavoidable in production:

• Scan quality degradation - OCR accuracy on clean scans runs 95-99%, but drops to 70-80% on degraded photocopies or re-scanned documents. In a 10,000-character document, that's up to 1,000 wrong characters.
• Table extraction complexity - the most commercially significant data (pricing, schedules, terms) usually lives in tables. Simple tables extract at 97-99% accuracy; real-world tables with merged cells, nested subtotals, and multi-page spans drop to 70-85%.
• Cross-reference context - a clause on page 47 can reference a defined term on page 3. Processing pages independently loses that link; processing the whole document at once can exceed context limits. The architecture has to solve for both.

When to Use LLM Extraction vs. Traditional OCR Alone

• Structured, standardised forms (fixed layout, known fields) - traditional OCR + template matching is often enough; an LLM adds cost without much benefit.
• Variable, unstructured documents (contracts, correspondence, mixed formats) - an LLM is needed to interpret meaning, not just position on the page.
• Anything feeding a decision or a report - always add confidence scoring and a human-review path, regardless of which extraction method you use.

Implementation Sketch

from azure.ai.formrecognizer import DocumentAnalysisClient
from azure.identity import DefaultAzureCredential

client = DocumentAnalysisClient(
    endpoint="https://my-doc-intel.cognitiveservices.azure.com/",
    credential=DefaultAzureCredential()
)

with open("document.pdf", "rb") as f:
    poller = client.begin_analyze_document("prebuilt-layout", f)
    result = poller.result()

for page in result.pages:
    for word in page.words:
        if word.confidence < 0.8:
            flag_for_review(word.content, page.page_number, word.confidence)

extraction_prompt = """Extract the requested fields from this document section.
Return JSON with a confidence score (0-1) and the source text for each field.

Fields to extract: {field_list}

Document text:
{chunk_text}
"""

response = llm_client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": extraction_prompt.format(field_list=fields, chunk_text=chunk)}],
    response_format={"type": "json_object"}
)

extraction = json.loads(response.choices[0].message.content)
for field, data in extraction.items():
    if data["confidence"] < 0.85:
        send_to_review_queue(field, data, document_id)

Platform Options

Key Takeaways

• OCR quality on degraded scans is usually the first bottleneck, not the LLM - profile your actual document population before assuming the LLM is the weak link.
• Confidence scoring at every stage (OCR word-level, LLM field-level) is what separates a production system from a demo.
• Tables carry the most commercially significant data and lose the most accuracy - invest in table-specific models before optimising the LLM prompt.
• Human-in-the-loop is a design choice, not a failure state. Route low-confidence extractions to review and capture corrections as future training signal.

Worked example: a Contract Extraction case study applying this pattern to a real diligence workflow, including production metrics and a build-vs-buy framework, is planned for a future release of this playbook.