Expand MOI RAG benchmark plans and parser evaluation - #15
Merged
Conversation
There was a problem hiding this comment.
🟡 Not ready to approve
The pipeline orchestrator and environment loading have confirmed functional gaps/inconsistencies (missing vlm pipeline option, dotenv empty-var handling) plus a redirect validation edge case and portability issues in docs that should be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR expands the MOI RAG benchmarking plan/spec documentation and significantly extends the local MatrixFlow parser + orchestration prototypes to support official MinerU cloud pipelines (precision/agent), a TaaS VLM OCR path, and end-to-end local pipeline coverage with focused tests.
Changes:
- Add/extend local parser support for official MinerU precision/agent pipelines, plus a TaaS VLM image-to-Markdown OCR route, including safety checks and tests.
- Update local RAG prototype to support rebuilding the vector table (for embedding dimension changes) and add an integration test covering the behavior.
- Add substantial Stage 1/three-stage benchmark planning/spec documentation, identity decision, and local runbook examples/.gitignore updates.
File summaries
| File | Description |
|---|---|
| rag/prototypes/local-matrixflow-rag/README.md | Updates --force semantics documentation (rebuild vector table). |
| rag/prototypes/local-matrixflow-rag/main.go | Changes vector table handling from truncate to drop+recreate on --force. |
| rag/prototypes/local-matrixflow-rag/main_test.go | Adds integration test validating rebuild when embedding dimension changes. |
| rag/prototypes/local-matrixflow-rag/data/questions-mineru-smoke.jsonl | Adds a MinerU smoke question case dataset row. |
| rag/prototypes/local-matrixflow-pipeline/test_pipeline.py | Adds CLI test ensuring parser pipeline + env-file propagation. |
| rag/prototypes/local-matrixflow-pipeline/README.md | Updates pipeline usage and documents parser pipeline routes and env behavior. |
| rag/prototypes/local-matrixflow-pipeline/pipeline.py | Adds .env loading and passes parser pipeline/env-file through to subprocesses. |
| rag/prototypes/local-matrixflow-pipeline/DESIGN.md | Updates design doc to describe explicit parser pipeline routing. |
| rag/prototypes/local-matrixflow-parser/taas_vlm.go | Introduces TaaS VLM OCR pipeline (image → Markdown → normalize via MatrixFlow). |
| rag/prototypes/local-matrixflow-parser/taas_vlm_test.go | Adds unit test coverage for TaaS VLM normalization flow. |
| rag/prototypes/local-matrixflow-parser/README.md | Documents MinerU pipelines, OpenXML, VLM setup, and local run commands. |
| rag/prototypes/local-matrixflow-parser/provider.go | Wires optional OpenXML/VLM/converter clients into the local provider. |
| rag/prototypes/local-matrixflow-parser/plan.go | Improves V3-native planning and makes missing dependencies explicit. |
| rag/prototypes/local-matrixflow-parser/parser.go | Adds pipeline routing (local/precision/agent/vlm), dependency marking, and doc annotation factoring. |
| rag/prototypes/local-matrixflow-parser/mineru.go | Implements official MinerU precision/agent pipeline client with download hardening + wget fallback. |
| rag/prototypes/local-matrixflow-parser/mineru_test.go | Adds tests for MinerU flows, ZIP safety, redirect safety, wget fallback behavior, env filtering. |
| rag/prototypes/local-matrixflow-parser/LOCAL_SERVICES_TODO.md | Notes MinerU official routing update; keeps older 302.AI notes as historical. |
| rag/prototypes/local-matrixflow-parser/cmd/local-matrixflow-parser/main.go | Adds CLI flags for pipeline/env-file/OpenXML/soffice/VLM and pipeline-aware plan output. |
| rag/prototypes/local-matrixflow-parser/cmd/local-matrixflow-parser/env.go | Adds .env discovery and selective env var resolution helper. |
| rag/prototypes/local-matrixflow-parser/cmd/local-matrixflow-parser/env_test.go | Adds unit tests for env resolution behavior. |
| rag/plans/todo/stage1-run-ledger-agent-todoplan.md | Adds detailed Stage 1 serial run ledger execution plan (agent version). |
| rag/plans/todo/rag-benchmark-catalog-and-layered-evaluation-plan.md | Adds benchmark catalog + layered evaluation plan. |
| rag/plans/todo/moi-rag-competitor-landscape.md | Adds competitor landscape doc marked as identity-polluted/do-not-cite. |
| rag/plans/todo/moi-rag-benchmark-three-stage-plan-v1.md | Adds three-stage benchmark execution plan. |
| rag/plans/todo/golden-and-metrics-spec-v0.4.md | Adds golden/metrics derived specification for v0.4 pilot. |
| rag/plans/rag-benchmark-primary-research-2026-08-04.md | Adds primary research + executable recommendations doc. |
| rag/plans/moi-official-primary-research-2026-08-04.md | Adds MOI official-source research summary and implications for benchmarking. |
| rag/plans/drafts/moi-product-positioning-and-local-test.md | Adds identity-polluted draft doc (do-not-cite). |
| rag/dify-rag-eval/.gitignore | Adds local multihop config ignore entry. |
| rag/decisions/moi-identity.md | Adds canonical identity decision for MOI. |
| rag/.gitignore | Expands ignore rules for local artifacts/runs/tmp; includes a duplicate entry. |
| rag/.env.example | Adds example .env template for MinerU/OpenXML/TaaS settings. |
Review details
- Files reviewed: 32/35 changed files
- Comments generated: 7
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Comment on lines
+96
to
+99
| "--parser-pipeline", | ||
| default="local", | ||
| choices=["local", "precision", "agent"], | ||
| help="local MatrixFlow parser or an official MinerU cloud pipeline", |
Comment on lines
+48
to
+51
| value = value.strip() | ||
| if len(value) >= 2 and value[0] == value[-1] and value[0] in {'"', "'"}: | ||
| value = value[1:-1] | ||
| environ.setdefault(key, value) |
Comment on lines
+645
to
+653
| func validateRedirectURL(from, to *url.URL) error { | ||
| if err := validateDownloadURL(to.String()); err != nil { | ||
| return fmt.Errorf("%w: %v", errUnsafeRedirect, err) | ||
| } | ||
| if !strings.EqualFold(from.Host, to.Host) { | ||
| return fmt.Errorf("%w: redirect host changed from %q to %q", errUnsafeRedirect, from.Host, to.Host) | ||
| } | ||
| return nil | ||
| } |
Comment on lines
26
to
31
| python3 pipeline.py \ | ||
| --input ../local-matrixflow-parser/data/sample.md \ | ||
| --input /absolute/path/document.pdf \ | ||
| --config ../local-matrixflow-rag/config.local.json \ | ||
| --parser-pipeline precision \ | ||
| --env-file /Users/muuushroom/gitrepos/moi-benchmark/rag/.env \ | ||
| --question "这个文档说明了什么？" \ |
Comment on lines
+105
to
+109
| Store the official MinerU token here: | ||
|
|
||
| ```text | ||
| /Users/muuushroom/gitrepos/moi-benchmark/rag/.env | ||
| ``` |
| @@ -0,0 +1,13 @@ | |||
| # Copy this file to /Users/muuushroom/gitrepos/moi-benchmark/rag/.env. | |||
Comment on lines
+14
to
+16
| tmp/ | ||
| ~$*.xlsx | ||
| tmp/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
go test ./...inprototypes/local-matrixflow-parsergo test ./...inprototypes/local-matrixflow-ragpython3 -m unittest discover -s . -p 'test*.py'inprototypes/local-matrixflow-pipelinegofmtandgit diff --checkScope notes
Downloaded public datasets,
runs/,tmp/, caches, generated artifacts, and Excel lock files remain local and ignored. The experiment record captures the planned/ongoing public benchmark evaluation without committing raw dataset payloads.