Agent Trajectory Cluster Audit

This repository contains the reproducibility code for the paper:

Auditing Unsupervised Trajectory Clustering in LLM-Agent Failure Analysis: A Multi-Dataset Validation Protocol.

The project audits a common claim in LLM-agent observability: that unsupervised clusters over agent trajectories align with failure labels. The paper does not propose a new failure taxonomy or a new clustering algorithm. It implements a four-part validation protocol for interpreting cluster-label alignment:

1. outcome-leakage audit,
2. stratified label validation against obvious task/framework confounds,
3. per-tag validation for multilabel annotations,
4. matched-K baselines against random, single-feature, and confound-oracle partitions.

The committed outputs are summary artifacts used for numeric checks. Raw datasets are not stored in this repository; they are downloaded from their public or gated upstream sources via the pinned downloader.

Repository Contents

configs/      Experiment YAML files.
src/          Dataset loaders, feature extraction, clustering, validation, reporting.
scripts/      Dataset download, robustness, ablation, case-study, and figure scripts.
outputs/      Summary CSV outputs and run manifests used by the paper.
figures/      Paper figures generated from the summary outputs.
main.py       Config-driven experiment entry point.

Excluded from the public release:

• raw datasets under data/,
• virtual environments and local caches,
• per-trajectory feature matrices and cluster-assignment CSVs,
• manuscript drafts and venue-specific submission files.

Environment

Python 3.10 or newer is required. The original experiments were run with Python 3.11.

Using uv:

uv venv
uv pip install -e ".[all]"

Using pip:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all]"

Download Data

The downloader recreates the expected directory layout under data/external/. Hugging Face revisions and git commits are pinned in scripts/download_external_datasets.py.

python scripts/download_external_datasets.py --datasets core

Notes:

• microsoft/AgentRx is gated on Hugging Face and requires prior access.
• If you want to run only the public non-gated downloads first:

python scripts/download_external_datasets.py --datasets core --skip-gated

Reproduce Main Runs

Run the three main benchmark configurations:

python main.py --config configs/experiments/aftraj.yaml
python main.py --config configs/experiments/agenterrorbench.yaml
python main.py --config configs/experiments/agentrx.yaml

Run the within-domain AFTraj configurations:

python main.py --config configs/experiments/aftraj_math.yaml
python main.py --config configs/experiments/aftraj_coding.yaml
python main.py --config configs/experiments/aftraj_agentic.yaml

Run the analysis scripts used for robustness and paper tables:

python scripts/seed_variance_analysis.py
python scripts/feature_group_ablation.py
python scripts/planning_phenotype_analysis.py
python scripts/agentrx_root_cause_analysis.py
python scripts/render_figures.py

Each run writes a run_manifest.json and run_manifest.yaml with the resolved configuration, config hash, software versions, and headline metrics.

Output Policy

This public release includes compact summary outputs such as cluster_label_validation.csv, cluster_baseline_comparison.csv, k_sensitivity.csv, seed-variance summaries, and run manifests.

It intentionally excludes raw trajectories and per-trajectory derived files (trajectory_features.csv, cluster_assignments.csv) so the repository stays small and source-focused. These files are regenerated by running the commands above after downloading the datasets.

Citation

If you use this code, cite the archived Zenodo release:

Lee, R., & Oh, T. (2026). Agent Trajectory Cluster Audit (v0.1.0). Zenodo. https://doi.org/10.5281/zenodo.20344064

The GitHub repository alone is not the archival citation target.

License

This code is released under the MIT License. See LICENSE.