📄 docling-batch-processor

A powerful batch document conversion tool that transforms PDF, Word, Excel, PPT, and more into clean, structured Markdown — with intelligent image and table handling.

---

✨ Features

• ✅ Batch Conversion – Convert PDF, Word, Excel, PPT, and more to clean Markdown
• ✅ Batch Chunking – Process Markdown files into Dify-ready formats
• 🔄 Concurrent Execution – Multi-threaded for faster throughput
• 🔍 Auto Discovery – Recursively scan directories for supported files
• 🖼️ Smart Image Handling – Extract images as files + embed Base64 fallback
• 📊 Table Optimization – Preserve layout with Markdown-compatible formatting
• 📈 Conversion Report – Detailed stats: success/failure, timing, image counts
• 🛡️ Robust Error Recovery – Gracefully handle corrupt or unsupported files
• ⚙️ Fully Configurable – Customize workers, output path, and Docling service URL

---

📁 Supported File Formats

Conversion (batch_convert)

Format	Extensions
PDF	.pdf
Microsoft Word	.docx, .doc
Excel	.xlsx, .xls
PowerPoint	.pptx
Plain Text	.txt
Web Documents	.html, .xml

Chunking (batch_chunk)

Format	Extensions
Markdown	.md

---

⚙️ System Requirements

• Python 3.9+
• Running Docling-serve instance (local or remote)

---

📦 Installation

1. Clone the Repository

git clone https://github.com/ethanrise/docling-batch-processor.git
cd docling-batch-processor

2. Install Dependencies

pip install -r requirements.txt

3. Start Docling-serve Service

Using Docker (Recommended)

docker run -p 9969:5001 quay.io/docling-project/docling-serve:latest

Using Docker Compose

Create docker-compose.yml:

version: '3.8'
services:
  docling-serve:
    image: quay.io/docling-project/docling-serve:latest
    ports:
      - "9969:5001"
    volumes:
      - ./uploads:/app/uploads

Then run:

docker-compose up -d

💡 Ensure port 9969 is accessible and not blocked by firewall.

---

▶️ Usage

Part 1: Batch Conversion (batch_convert.py)

Convert various document formats to Markdown:

# Convert specific files
python batch_convert.py file1.pdf file2.docx

# Convert all supported files in a directory
python batch_convert.py -d /path/to/documents

# Specify custom output directory
python batch_convert.py file1.pdf -o ./results

# Increase concurrency & use custom service URL
python batch_convert.py -d ./docs -o ./results --workers 5 --url http://remote-server:9969/v1/convert/file

Part 2: Batch Chunking (batch_chunk.py)

Process Markdown files into Dify-ready format:

# Process all Markdown files in a directory
python batch_chunk.py -d /path/to/markdown/files

# Specify custom output directory
python batch_chunk.py -d ./markdown_files -o ./dify_ready

# Use custom API endpoint
python batch_chunk.py -d ./docs --url http://remote-server:9969/v1/chunk/hybrid/source

📌 Command-Line Arguments

batch_convert.py

Argument / Flag	Description	Default
input_files	List of input file paths	(required if -d not used)
-d, --directory	Input directory to scan recursively	—
-o, --output	Output directory	Parent of first input file
--workers	Number of concurrent workers	3
--url	Docling service endpoint	http://localhost:9969/v1/convert/file

batch_chunk.py

Argument / Flag	Description	Default
-d, --directory	Input directory containing Markdown files	(required)
-o, --output	Output directory	{input_dir}/../dify_ready
--url	Document chunking service endpoint	http://127.0.0.1:9969/v1/chunk/hybrid/source

---

📂 Output Structure

After Conversion (batch_convert.py)

Each converted file generates:

output/
├── document.md                 # Clean Markdown output
├── document_images/            # Extracted images (PNG/JPG)
│   ├── image_20260129_001.png
│   └── image_20260129_002.jpg
└── conversion_report.txt       # Summary report

After Chunking (batch_chunk.py)

Each Markdown file generates a processed version:

output/
├── document_processed.txt      # Clean, chunked content ready for Dify
└── processing_report.txt       # Summary report

---

📊 Conversion Report

The conversion_report.txt includes:

• Total files processed, success/failure counts
• Per-file processing time
• Number of extracted images
• Error details for failed conversions

Example snippet:

✅ Successfully converted: 12 files
❌ Failed: 2 files
⏱️ Total time: 42.3s
📸 Total images extracted: 37

---

🏗️ Project Structure

docling-batch-processor/
├── batch_convert.py            # CLI entry point for conversion
├── batch_chunk.py              # CLI entry point for chunking
├── core/
│   ├── batch_converter.py      # Orchestrates the full pipeline
│   ├── docling_client.py       # HTTP client for Docling API
│   ├── file_validator.py       # Validates input files
│   ├── image_processor.py      # Handles image extraction & saving
│   ├── table_processor.py      # Optimizes table formatting
│   └── output_manager.py       # Manages output files & report
└── requirements.txt

Module Roles

• batch_converter: Main controller that coordinates all components
• docling_client: Wraps API calls to Docling-serve
• image_processor / table_processor: Post-process conversion results
• output_manager: Handles file I/O and report generation
• markdown_processor: Main controller for chunking pipeline

---

❓ Common Issues

🔌 Connection Refused

• ✅ Is Docling-serve running? Check with docker ps
• ✅ Is the URL correct? Default: http://localhost:9969/v1/convert/file
• ✅ Is the port open? Try curl http://localhost:9969/health

📄 Invalid File Format

• Ensure files aren’t corrupted
• Extension must match actual format (e.g., don’t rename .zip to .docx)

💥 Out of Memory

• Reduce concurrency( batch_convert.py): --workers 1
• Process smaller batches

---

🤝 Contributing

Contributions are welcome! Please:

1. Open an issue to discuss feature requests or bugs
2. Fork the repo and submit a PR with clear commit messages
3. Add tests if applicable

See CONTRIBUTING.md (optional but recommended).

---

📬 Author & Contact

• Email: thanrise.ai@gmail.com
• GitHub: @ethanrise
• LinkedIn: linkedin.com/in/ethanrise

💬 Feel free to reach out for collaboration or feedback!