DocIntel AI — Multi-Modal Document Understanding System

Layout-aware document classification, information extraction, and visual QA using LayoutLMv3.

---

Architecture

PDF/Image Upload
      │
      ▼
 Tesseract OCR ──────────────────────┐
      │                              │
      ▼                              │
LayoutLMv3 Classifier          words + boxes
(Document Type)                      │
      │                              ▼
      │                   LayoutLMv3 Extractor
      │                   (Named Entity Recognition)
      │                              │
      └──────────────────────────────┤
                                     ▼
                            DocVQA Pipeline
                          (Question Answering)
                                     │
                                     ▼
                           Streamlit Interface

Features

Feature	Model	Dataset
Document Classification	LayoutLMv3	RVL-CDIP (400K docs, 16 classes)
Information Extraction	LayoutLMv3	FUNSD (form understanding)
Document QA	impira/layoutlm-document-qa	DocVQA
OCR	Tesseract OCR	—

Quick Start

1. Install dependencies

# Install Tesseract (system)
sudo apt-get install tesseract-ocr  # Linux
brew install tesseract              # macOS

# Install Python dependencies
pip install -r requirements.txt

2. Run the app (with base models — no fine-tuning needed for QA)

streamlit run app.py

3. Fine-tune on Kaggle (for classification + extraction)

1. Upload notebooks/train_classifier.ipynb to Kaggle
2. Enable GPU (T4 — free tier)
3. Run all cells
4. Download checkpoint to models/classifier_checkpoint/
5. Update configs/config.yaml → checkpoint_path

Project Structure

docintel/
├── app.py                      # Streamlit UI
├── requirements.txt
├── configs/
│   └── config.yaml             # All hyperparameters & paths
├── src/
│   ├── ocr/
│   │   └── pipeline.py         # Tesseract + pdf2image
│   ├── classification/
│   │   └── classifier.py       # LayoutLMv3 for doc classification
│   ├── extraction/
│   │   └── extractor.py        # LayoutLMv3 for NER/IE
│   ├── qa/
│   │   └── document_qa.py      # DocVQA pipeline
│   └── utils/
│       └── helpers.py          # Shared utilities
├── notebooks/
│   └── train_classifier.ipynb  # Kaggle training notebook
└── models/
    ├── classifier_checkpoint/  # Fine-tuned weights (after training)
    └── extractor_checkpoint/   # Fine-tuned weights (after training)

Training Strategy

Local machine (16GB RAM, Intel Iris Xe):

• Code development, testing, inference
• Run streamlit run app.py with pretrained models

Kaggle (Tesla T4 GPU):

• Fine-tune LayoutLMv3 on RVL-CDIP (~6-8 hours for full training)
• Fine-tune LayoutLMv3 on FUNSD (~2-3 hours)
• Export checkpoints → store in models/

Hugging Face Spaces:

• Deploy app.py + push model weights to HF Hub
• Update config.yaml hf_hub_model fields

Evaluation Targets

Task	Metric	Target
Classification	Accuracy	>90%
Classification	Weighted F1	>88%
Extraction	Entity F1	>75%
QA	Exact Match	>65%
QA	F1	>75%

Datasets

• RVL-CDIP: aharley/rvl_cdip on HF Hub (~35GB)
• FUNSD: nielsr/funsd on HF Hub
• DocVQA: docvqa on HF Hub

Tech Stack

PyTorch • Hugging Face Transformers • LayoutLMv3 • Tesseract OCR
Streamlit • Plotly • Pandas • NumPy • OpenCV • pdf2image

---

Built a ML project demonstrating Computer Vision, NLP, OCR, and Transformer-based document understanding.