Home

PathwAI

Wrap a fallible LLM in search and a verifier, and prove it plans like the optimum.

PathwAI turns an LLM into a deliberate planner. A language-model proposer estimates how far each state is from the goal; a best-first search uses that estimate as a heuristic; and a verifier rejects illegal or looping moves so the agent recovers from bad suggestions instead of executing them blindly. Every plan is scored against a built-in A* optimum, so the claims are measured, not asserted.

The whole benchmark — three planning domains, five planners, a null ablation — runs green in CI with no API keys and no model downloads using a deterministic stub proposer. Real LLM backends (Ollama, OpenAI) are opt-in via pip extras.

Architecture

flowchart LR
    subgraph Domains["Domain (env-selectable)"]
        GW[GridWorld]
        BW[Blocksworld]
        DL[Delivery]
    end
    subgraph Proposer["LLM proposer (env-selectable)"]
        STUB[stub\nnoisy heuristic · offline]
        RND[random\nablation]
        REAL[ollama / openai\noptional extras]
    end
    Domains --> STATE[State + applicable actions]
    STATE --> Proposer
    Proposer --> EST[cost-to-go estimate]
    EST --> SEARCH[Best-first search\npriority = g + estimate]
    STATE --> SEARCH
    SEARCH <--> VERIFY[Verifier\nlegal? not revisited?]
    SEARCH --> PLAN[Plan]
    Domains -.optimal A*.-> SCORE[Score vs. ground-truth optimum]
    PLAN --> SCORE
    SCORE --> M[solve rate · optimality ratio · expansions]

Loading

Quick start

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytest -q                    # 77 tests
pathwai compare --domain gridworld --seed 3

Wiki pages

Page	What it covers
Architecture	Domain representation, proposer, verifier, search engine, ablation design
Evaluation	Benchmark setup, results table, ablation, reproduce commands
Configuration	Env vars, backend matrix, .env.example
Development	Setup, test commands, adding domains and proposers, code layout