Multi-Document Corpus Indexing

[krishna231122-del]

📝 Overview

This PR introduces Multi-Document Corpus Indexing, allowing users to merge multiple individual document trees into a single, unified knowledge base.

By running multiple structures through the new Corpus Builder, Agentic RAG systems can now navigate and reason across an entire repository of documents simultaneously from a single root node, rather than being limited to a single document's context.

🛠️ Changes Made

• Corpus Builder (pageindex/corpus_builder.py):

  • Added a new utility that mounts multiple *structure.json files as child nodes under a unified "Corpus" root node.
  • Automatically re-numbers all node_ids sequentially across the entire merged tree to prevent cross-document ID collisions and ensure reliable traversal.

• CLI Enhancements (run_pageindex.py):

  • Added the --corpus_files argument to specify a list of pre-compiled JSON structure files to merge.
  • Added the --corpus_name argument to allow customizing the title of the root node (defaults to "Corpus").
  • Invoking the CLI with these flags bypasses standard document parsing and generates the merged [corpus_name]_structure.json.

• Dummy Files & Gitignore:

  • Added examples/documents/test_doc2.txt for integration testing.
  • Updated Gitignore where required to prevent generated test/output files from being committed.

🧪 Testing

• Successfully compiled two independent Markdown structures:

  • test_doc_converted
  • test_doc2_converted

• Merged them using:

python3 run_pageindex.py \
  --corpus_files \
  results/test_doc_converted_structure.json \
  results/test_doc2_converted_structure.json \
  --corpus_name "Test Knowledge Base"