Ed4All

Turn a textbook PDF into an accessible, course-ready package — semantic HTML, weekly modules, learning objectives, assessments, and a knowledge graph — in a single command.

Building a usable knowledge package from raw source material is weeks of manual work: extracting content, tagging it with learning science metadata, structuring it into pedagogically sound modules, writing aligned assessments, and validating accessibility. Ed4All runs that pipeline end-to-end, and everything it produces is WCAG 2.2 AA compliant by default.

What you get

Point Ed4All at a textbook PDF (or a directory of PDFs) and a course name, and it produces:

• Accessible HTML — semantic structure, proper heading hierarchy, alt text for images, ARIA landmarks, keyboard navigation, dark mode, and full WCAG 2.2 AA coverage.
• An LMS-ready IMSCC package — weekly modules with pages, activities, self-checks, summaries, and discussions, importable into Brightspace, Canvas, Blackboard, or Moodle.
• Bloom's-aligned learning objectives — per module and per page, each tagged with a cognitive domain and linked back to the source content.
• A knowledge graph — chunked content with key terms, misconceptions, learning-outcome references, and a typed concept graph covering taxonomic and pedagogical structure.
• A reusable archive — the course is indexed into a local knowledge repository you can query with hybrid retrieval (BM25 fused with dense vector search via reciprocal rank fusion), filter by concept or objective, and reuse across courses.
• Training data, and optionally a course-tuned model — instruction pairs for supervised fine-tuning (SFT) and preference pairs for direct preference optimization (DPO), synthesized from the course's own chunks and assessments. Pass --with-training and Ed4All goes one step further: a LoRA fine-tune (bf16 PEFT) of a small language model on those pairs, with promotion gates, a base-vs-adapter eval matrix, and a full provenance model card.

Every chunk carries its Bloom's level, content type, key terms, misconceptions, and the original PDF region it came from, so downstream LLMs can ground their answers in cited source material.

Who it's for

• Instructors and instructional designers producing online courses from textbook source material at scale.
• Accessibility teams remediating document libraries to WCAG 2.2 AA compliance.
• EdTech and ML teams building AI tutors, RAG assistants, or domain-adapted language models that need pedagogically structured training data.
• Researchers studying retrieval quality, assessment generation, or learning-science-aligned content representations.

How it works

textbook-to-course runs up to 24 phases end-to-end. Each phase checkpoints on completion (a failed or stopped run resumes where it left off), quality gates validate the artifacts between phases, and GPU model seats are started and stopped automatically so only the models a phase needs are ever resident.

#	Phase	What it does	GPU workload
1	semantik_conversion	PDF → accessible semantic HTML (layout extraction + image alt text)	OCR + vision models
2	heading_judge	Re-levels ambiguous heading levels via a large-model judge (runs by default; no-op on born-digital corpora)	Large model
3	staging	Stages converted HTML for course generation	—
4	chunking	Emits the deterministic source chunkset	—
5	objective_extraction	Parses staged HTML into the textbook structure (chapters, sections, blocks)	—
6	source_mapping	Maps source blocks to course module pages	—
7	course_planning	Synthesizes terminal + component learning objectives from the structure	Large model
8	concept_extraction	Builds the typed concept / knowledge graph	Large model
9	content_generation	Single-pass content authoring (alternative to 10–12)	Large model
10	content_generation_outline	Two-pass tier 1: terse per-block outlines	Large model
11	inter_tier_validation	Structural validators over the outline tier (no LLM)	—
12	content_generation_rewrite	Two-pass tier 2: full HTML block bodies	Large model
13	assessment_synthesis	Grounded quizzes, assignments, and discussions as QTI/IMS XML (optional)	Large model
14	post_rewrite_validation	The largest gate set: prose entailment, claim support, block quality	Local validators (NLI + embeddings)
15	packaging	Packages the course as IMSCC	—
16	imscc_chunking	Emits the post-packaging retrieval chunkset	—
17	trainforge_assessment	Generates assessments from the packaged course (optional)	—
18	training_synthesis	Synthesizes SFT instruction pairs and DPO preference pairs from the course chunks and assessments (optional)	Large model
19	libv2_archival	Archives all artifacts to the local course library	—
20	vector_indexing	Builds the per-course vector index, enabling semantic and hybrid-RRF retrieval so the course is immediately askable	Local embeddings
21	training	LoRA fine-tune (SFT + DPO) of a small language model on the course's instruction and preference pairs (bf16 PEFT, licensing preflight, full provenance card) (opt-in)	Training (exclusive — all serving seats stopped)
22	post_training_validation	Promotion gates on the trained adapter — eval thresholds and CURIE-family completeness (opt-in)	—
23	evaluation	5-layer × 3-tier eval matrix vs the base model, adapter audit, promote / hold / reject decision (opt-in)	Local eval models
24	finalization	Final validation and training-data export	—

Phases 18 and 21–23 are the training path. Phase 18 synthesizes the corpus — SFT instruction pairs and DPO preference pairs, grounded in the course's own chunks and assessments — and phases 21–23 fine-tune a small language model on it with LoRA, then gate and evaluate the result. The fine-tune is off by default (a training run takes hours and needs the whole GPU), so phases 21–23 run only when you pass --with-training; --skip-training wins if you pass both. A default build skips them and still finishes at finalization.

Training an already-archived course without rebuilding it stays a separate follow-on workflow: ed4all run trainforge_train --course-name <slug>.

GPU workload legend: OCR + vision models — the lightweight extraction and alt-text seats; Large model — the main authoring/judging model seat; Local validators / embeddings — in-process models (no serving seat); — — deterministic CPU work. Seats in different groups are swapped at phase boundaries automatically, with health and coherence checks at every start.

Quick start

Requires Python 3.10+. Optional system tools (tesseract-ocr, poppler-utils) improve extraction on scanned or image-heavy PDFs.

git clone https://github.com/mdmurphy822/Ed4All.git
cd Ed4All
pip install -e ".[full]"

# Convert a textbook PDF into a full course package
ed4all run textbook-to-course --corpus my_textbook.pdf --course-name MY_COURSE_101

[full] installs the CPU-light surface: PDF-to-HTML conversion, the MCP server, the dev/test toolchain, and the GUI. It deliberately leaves out the two heavy ML extras so a default install never pulls multi-GB GPU wheels:

Extra	Adds	When you need it
embedding	sentence-transformers + torch	Semantic and hybrid-RRF retrieval, plus the statistical-tier content validators. Without it those validators degrade to warnings and retrieval falls back to BM25 alone.
training	torch + transformers + trl + peft + bitsandbytes	LoRA fine-tuning (SFT + DPO) of a course-pinned SLM adapter (ed4all run trainforge_train). Requires a GPU.

Install them alongside [full] only when needed:

pip install -e '.[full,embedding]'   # + semantic retrieval / statistical validators
pip install -e '.[full,training]'    # + SLM fine-tuning (GPU)

By default Ed4All runs in local mode — no API key required. To route orchestration through the Anthropic API instead, set ANTHROPIC_API_KEY and add --mode api.

Content generation is model-agnostic: every authoring, synthesis, and answer provider speaks the OpenAI-compatible API, so any local model server (Ollama, vLLM, llama.cpp) or hosted endpoint plugs in with just a base URL, an API key, and a model name — configuration, not code. Swap models or providers per task without touching the pipeline.

That single command runs the full pipeline — accessibility conversion, objective synthesis, course planning, module generation, IMSCC packaging, knowledge-graph building, and archival. The IMSCC file lands in Courseforge/exports/, and the searchable archive lands in LibV2/courses/.

Prefer a GUI?

Ed4All ships a browser-based control panel for the whole pipeline — upload PDFs, manage API keys and environment, choose which model runs each task (including local Ollama and vision/VLM models), edit course topics and learning objectives, launch a full run or a single stage with live logs, and query the knowledge base — no command line required.

# One click: builds a virtualenv, installs, starts the server, opens your browser
./run-gui.sh           # macOS / Linux
run-gui.bat            # Windows (double-click)

If you already ran pip install -e ".[full]" above (it includes the GUI), just launch it directly:

ed4all gui             # serves http://127.0.0.1:8077

To manage pipeline runs from the GUI's Assistant panel — ask what a run is doing, prepare and launch builds, resume or stop them — launch with the assistant seat enabled:

ED4ALL_ASSISTANT_AUTOSTART=1 ed4all gui   # assistant panel may start its local model seat on demand

The assistant picks its model dynamically: if a larger local model seat is already serving (for example while a build is running), it answers through that seat and the panel shows which model replied; otherwise it brings up its own small default seat. Details in The built-in assistant.

Prefer containers?

A two-service Docker Compose stack serves the GUI on http://localhost:8077 alongside a local model backend — no API key, no command line:

docker compose up -d                                          # build + start the stack
docker compose exec ollama ollama pull qwen2.5:7b-instruct-q4_K_M   # pull the answer model (one-shot)
# then open http://localhost:8077, upload a PDF, and run the pipeline

CPU-only by default; optional NVIDIA GPU support, volume layout, and remote-deploy auth are covered in docs/operations/docker.md.

Full GUI guide — the six panels, settings and secret handling, model routing, retrieval, and how Claude Code sessions can drive it: gui/README.md.

Other useful commands:

ed4all run --help                                     # List workflows and flags
ed4all run textbook-to-course --dry-run ...           # Plan only, no execution
ed4all run textbook-to-course --resume <run_id>       # Resume an interrupted run
ed4all stop <run_id>                                  # Checkpoint and pause at the next unit boundary
ed4all list-runs                                      # Show recent runs

Running a full production build

The quick-start command above is the happy path and needs no configuration. A full multi-hour build against local model seats — seat topology, the phase sequence and what each phase produces, how to read gate outcomes, and the resume/stop procedure — is covered in one place: docs/operations/full-run-playbook.md.

Environment for such a run starts from run-env.example.sh. Read its hardware-profile section before copying any concurrency, batch-size, or GPU-lifecycle setting: those values are tuned for a single-GPU large-memory host and will exhaust VRAM on a small card unedited.

The built-in assistant

Ed4All ships a local AI assistant for operating the pipeline — ask it what a run is doing, why a gate failed, or have it start the next build:

ed4all assistant                          # interactive chat
ed4all assistant --once "why did my last run fail?"
ed4all assistant --debug                  # open a session pre-loaded with the
                                          # most recent failure's diagnostics

It speaks through a fixed set of typed tools — run status and gate reports, bounded log tails, ed4all doctor diagnostics, build-cost and quality reports, course library inspection, grounded Q&A against any built course's index, and a small set of guarded actions (start / resume / stop runs, seat start/stop, support bundles). It can also walk you through model-seat setup: it audits your seat-swap environment variables, discovers running model containers, and generates a ready-to-source configuration block. It never gets shell access or free-form file access — everything flows through validated tools, so it can help without being able to hurt.

The assistant is model-agnostic: it talks to any OpenAI-compatible local endpoint (ED4ALL_ASSISTANT_BASE_URL / ED4ALL_ASSISTANT_MODEL), restricted to localhost by design. The reference deployment — what it is tuned and tested against — is NVIDIA's Nemotron Nano (the NeMo model family) served with vLLM; set ED4ALL_ASSISTANT_AUTOSTART=1 to let the CLI bring that seat up on demand, and ED4ALL_ASSISTANT_DEBUG_ON_FAILURE=1 to have failed pipeline runs print the exact debug command to investigate them. The same assistant is available as a chat panel in the GUI (see Prefer a GUI? for launching it with the assistant enabled).

Dynamic model selection. When several local seats are registered (ED4ALL_SEAT_BASE_URLS), the assistant probes them in ED4ALL_ASSISTANT_SEAT_PRIORITY order (default spark-super,spark-nano) and answers through the first one that is live — so while a build has a large model seat loaded, the assistant (CLI and GUI panel alike) automatically upgrades to it, sharing the seat with the running pipeline. If nothing is live, it starts only its own small default seat (ED4ALL_ASSISTANT_SEAT), never the large one.

Managing multi-run pipelines. The assistant can also organize builds end to end: list your input corpora, arrange the environment for a run as a validated configuration overlay (it prepares settings from a curated allowlist — it can never write scripts or code), launch a build, watch its phases, resume a paused run, stop one gracefully, and compile structured error reports for a human (or a Claude Code session) to review whenever something needs an actual fix:

ed4all assistant --campaign        # interactive run-management session
ed4all assistant --campaign-tick   # one non-interactive monitoring pass (for schedulers)

What's inside

Ed4All is organised around four components that each do one job well, plus the glue that orchestrates them:

• SemantiK turns PDFs into accessible, semantic HTML using a license-clean extraction cascade (text layer, layout analysis, OCR, and learned structure/semantic classification) with per-block source provenance.
• Courseforge generates structured weekly course modules with learning objectives, assessments, interactive components, and rich machine-readable metadata, and packages them as IMSCC.
• Trainforge extracts content from the course package into pedagogically tagged chunks, builds a typed concept graph, and generates Bloom's-aligned assessments. It also synthesizes the SFT instruction pairs and DPO preference pairs, and owns the LoRA training, evaluation, and promotion workflow for course-pinned adapters.
• LibV2 is the archive and retrieval layer: a flat-storage course repository with hybrid BM25 + vector retrieval fused by reciprocal rank fusion (RRF), metadata filters, and cross-course concept indexes.

Supporting directories: MCP hosts the orchestrator and tool server, cli is the ed4all command line entry point, and lib holds shared validators and ontology helpers. Output artefacts land under Courseforge/exports/, LibV2/courses/, and runtime/training-captures/.

Going deeper

• Developer guide and orchestration protocol: CLAUDE.md
• Component guides: SemantiK/CLAUDE.md, Courseforge/CLAUDE.md, Trainforge/CLAUDE.md, LibV2/CLAUDE.md
• Ontology and schemas: schemas/ONTOLOGY.md

License

Apache License 2.0 — see LICENSE and NOTICE.