AI Form Autofill from Documents: Donut, LLM & PyTorch

Note: when a document is scanned, standard OCR outputs flat text without structure. An address might end up in the "Full Name" field, and a date in the notes. This situation is typical in accounting, where manual data entry takes hours. We solve this problem with a Document Understanding pipeline — it not only recognizes characters but understands where each piece belongs. In practice, this speeds up data entry 5–10 times and reduces errors by 70%. For a team of five operators, savings can reach 1.5 million rubles per year. The system pays for itself in 3–6 months through reduced labor costs.

Problems We Solve

• Unstructured text after OCR. Classic Tesseract or Google Vision produce raw text with coordinates but don't know that "Ivanov" is a last name and "123456" is a passport number. We use LayoutLM v3, which analyzes both text and block geometry.
• Different document templates. An invoice from one supplier differs from another, and a passport changes page layouts. Our pipeline classifies the document type and applies a specialized extractor.
• Low accuracy on complex documents. Invoices with tables, handwritten notes, low-quality scans — all break regular OCR. We add an LLM with vision that understands context and recovers missing information.

How AI Autofill from Documents Speeds Up Data Entry?

The core is a three-stage pipeline: classification → extraction → mapping. For standard types (passport, TIN) we use LayoutLM trained on thousands of labeled samples. For non-standard ones — multimodal LLM that processes the image and returns JSON with fields.

from typing import Any
from dataclasses import dataclass

@dataclass
class FormField:
    name: str           # имя поля в целевой форме
    value: Any          # извлечённое значение
    confidence: float   # уверенность 0..1
    source_location: str | None = None  # откуда извлечено

class DocumentToFormPipeline:
    """
    Пайплайн: документ → поля формы.
    Шаги:
    1. Классификация типа документа
    2. OCR + layout analysis
    3. NER/IE для извлечения полей
    4. Маппинг на поля целевой формы
    5. Валидация и нормализация
    """
    def __init__(
        self,
        document_classifier,   # модель классификации типа документа
        extractors: dict,      # {doc_type: extractor}
        form_mapper: dict,     # {doc_field: form_field} маппинг
        validators: dict       # {form_field: validator_fn}
    ):
        self.classifier = document_classifier
        self.extractors = extractors
        self.form_mapper = form_mapper
        self.validators = validators

    def process(self, document_image) -> dict[str, FormField]:
        # Шаг 1: определяем тип документа
        doc_type = self.classifier.predict(document_image)

        if doc_type not in self.extractors:
            raise ValueError(f'Unsupported document type: {doc_type}')

        # Шаг 2-3: извлечение полей
        extractor = self.extractors[doc_type]
        raw_fields = extractor.extract(document_image)

        # Шаг 4: маппинг
        form_fields = {}
        mapping = self.form_mapper.get(doc_type, {})

        for doc_field, value in raw_fields.items():
            if doc_field in mapping:
                form_field_name = mapping[doc_field]
                form_fields[form_field_name] = FormField(
                    name=form_field_name,
                    value=value.get('text'),
                    confidence=value.get('confidence', 0.0),
                    source_location=doc_field
                )

        # Шаг 5: валидация
        for field_name, field in form_fields.items():
            if field_name in self.validators:
                try:
                    field.value = self.validators[field_name](field.value)
                except Exception as e:
                    field.confidence *= 0.5   # снижаем уверенность при ошибке валидации

        return form_fields

Hybrid LayoutLM and LLM — Advantage Over Classic OCR

Classic OCR doesn't understand semantics: the word "Moscow" could be a city or a street name. LLM with vision (e.g., Claude 3.5 or GPT-4o) analyzes the entire document at once: it sees field locations, related headers, and tables. This boosts accuracy by 15–20% on complex layouts. We use LLM as a fallback for rare document types. OCR alone does not solve structure understanding.

import anthropic
import base64
from pathlib import Path

def extract_form_fields_llm(
    document_path: str,
    form_schema: dict,    # JSON Schema целевой формы
    model: str = 'claude-opus-4-5'
) -> dict:
    """
    Мультимодальный LLM как универсальный Document Understanding engine.
    form_schema: {'field_name': {'type': 'str', 'description': '...', 'required': bool}}
    """
    client = anthropic.Anthropic()

    # Загружаем документ как base64
    with open(document_path, 'rb') as f:
        doc_b64 = base64.standard_b64encode(f.read()).decode()

    ext = Path(document_path).suffix.lower()
    media_type = {
        '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg',
        '.png': 'image/png',  '.pdf': 'application/pdf'
    }.get(ext, 'image/jpeg')

    # Формируем описание схемы формы
    schema_desc = '\n'.join([
        f'- {name}: {info["description"]} ({"обязательное" if info.get("required") else "опциональное"})'
        for name, info in form_schema.items()
    ])

    message = client.messages.create(
        model=model,
        max_tokens=2048,
        messages=[{
            'role': 'user',
            'content': [
                {
                    'type': 'image',
                    'source': {
                        'type': 'base64',
                        'media_type': media_type,
                        'data': doc_b64
                    }
                },
                {
                    'type': 'text',
                    'text': f"""Извлеки из документа следующие поля для заполнения формы:

{schema_desc}

Верни результат строго в формате JSON:
{{
  "field_name": {{"value": "...", "confidence": 0.0-1.0, "not_found": false}},
  ...
}}

Если поле не найдено в документе, укажи "not_found": true.
Для confidence: 1.0 = точно уверен, 0.5 = сомневаюсь."""
                }
            ]
        }]
    )

    import json
    try:
        return json.loads(message.content[0].text)
    except json.JSONDecodeError:
        # Извлекаем JSON из текстового ответа
        import re
        match = re.search(r'\{.*\}', message.content[0].text, re.DOTALL)
        return json.loads(match.group()) if match else {}

How We Ensure Extraction Accuracy?

We combine three verification levels:

• LayoutLM outputs confidence for each field based on trained model.
• LLM additionally checks logical consistency (e.g., birth date not later than today).
• User-in-the-loop: fields with confidence < 0.85 are highlighted in the UI for manual review. This is standard practice in enterprise solutions.

def prepare_form_ui_state(
    form_fields: dict,
    confidence_threshold: float = 0.85
) -> dict:
    """
    Подготовка состояния формы для UI:
    - Поля с высокой уверенностью — автозаполнены
    - Поля с низкой — помечены для проверки
    - Обязательные не найденные — ошибка
    """
    ui_state = {}
    for field_name, field in form_fields.items():
        status = 'autofilled'
        if field.value is None:
            status = 'not_found'
        elif field.confidence < confidence_threshold:
            status = 'needs_review'

        ui_state[field_name] = {
            'value': field.value,
            'status': status,
            'confidence': field.confidence,
            'editable': status != 'autofilled' or True  # всегда редактируемо
        }
    return ui_state

Approach Comparison Table

Timelines

What's Included

• API documentation (Swagger, Postman collection) and pipeline description.
• Model training on your samples (up to 100 documents free).
• Trial period — 2 weeks with engineer support.
• Integration — ready modules for 1C and Bitrix24.
• Maintenance — 95% accuracy guarantee for 3 months, SLA on bug fixes.

Our team has 10 years of experience in Computer Vision and NLP and has delivered over 50 Document Understanding projects. We guarantee the system fits your landscape without delays. Request a demo version for testing on your documents. Contact us for a project evaluation — we'll prepare a prototype in 2 weeks.