Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
da86645
Add Streamlit demo app with pipeline toggle.
CHDev2116 May 6, 2026
3b40809
Simplify README and add container setup files.
CHDev2116 May 6, 2026
ded5a30
docs: align branding, collapsible README, and DX section
CHDev2116 May 9, 2026
1aebeda
Add MIT license, contributing guide, and public docs.
CHDev2116 May 14, 2026
6cd2f27
Stop tracking non-Architecture docs; ignore local interview materials.
CHDev2116 May 14, 2026
97052de
Lint entire src/ and tests in CI; fix Ruff findings.
CHDev2116 May 14, 2026
f8c5df1
Align tests and README with eval/engine coverage; defer roadmap items.
CHDev2116 May 14, 2026
44986d7
Add Streamlit demo screenshot to assets and README.
CHDev2116 May 14, 2026
d6c324c
Rename demo screenshot to assets/streamlit-comparison.png.
CHDev2116 May 14, 2026
17a55bf
Align packaging, fix verify script, and correct Docker llama/OpenCV s…
CHDev2116 May 14, 2026
f11e233
Single-source deps in pyproject; Streamlit import path and logging.
CHDev2116 May 14, 2026
e6973a2
Replace print with logging across Python sources.
CHDev2116 May 14, 2026
3ec2104
Gitignore generated logs JSON; drop stray committed error report.
CHDev2116 May 14, 2026
1d997f5
Gitignore JSON under logs/ from local pipeline runs.
CHDev2116 May 14, 2026
0fbe03f
Centralize CLI log format with timestamp and level.
CHDev2116 May 14, 2026
e044aa1
Use Pillow get_flattened_data; enforce minimum coverage in pytest.
CHDev2116 May 14, 2026
0be28c3
Add mypy to CI and dev; raise coverage floor; fix Streamlit import gr…
CHDev2116 May 14, 2026
15c4926
README: document mypy and coverage gate in CI/local checks.
CHDev2116 May 14, 2026
92b1864
Ignore interview language prep doc and drop unused yfinance dependency.
CHDev2116 May 19, 2026
6ea0c9c
Add async batch inference and parallel metrics for live model runs.
CHDev2116 May 22, 2026
3eccdc5
Upgrade loopback planner to be production-observable and fail-fast.
CHDev2116 May 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .cursorrules
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Repository / agent behavior
- Do not add, commit, or suggest pushing interview scripts, LinkedIn drafts, or other personal narrative markdown under `docs/` unless the user explicitly asks. Technical docs (e.g. `docs/Architecture.md`) are fine.

# Code Review Standards
- All Python async functions must include detailed logging.
- Test scripts must include API timeout handling.
Expand Down
31 changes: 31 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Git metadata
.git
.gitignore

# Python cache / virtual env
__pycache__/
*.py[cod]
*.pyo
*.pyd
venv/
env/
.env

# Test / coverage artifacts
.coverage
coverage.xml
htmlcov/
.pytest_cache/

# Local outputs and generated data
results/
logs/

# Local editor / OS files
.DS_Store
.vscode/
.idea/

# Large local assets not required for image build
test_images/
assets/
18 changes: 6 additions & 12 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,15 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest pytest-cov ruff
pip install -e ".[dev]"

- name: Lint
run: ruff check src tests app.py test_connection.py

- name: Type check (mypy)
run: |
ruff check \
src/ai_quality_agent.py \
src/eval \
src/models/inference_adapter.py \
src/util/failure_memory.py \
src/util/monitor_performance.py \
src/agent/orchestrator.py \
src/engine/image_processor.py \
src/test_failure_memory_retrieval.py \
tests
mypy --explicit-package-bases src
MYPYPATH=src mypy --explicit-package-bases app.py test_connection.py

- name: Unit tests with coverage
run: |
Expand Down
10 changes: 9 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ results/**/*.json
# Local run artifacts (loopback cache, vector DB, coverage)
results/loopback_cache/
results/failure_memory_db/
logs/**/*.json
.coverage
coverage.xml
htmlcov/
Expand All @@ -30,4 +31,11 @@ test_images/
*.gguf

# Packaging artifacts
*.egg-info/
*.egg-info/

# Local-only / interview prep (not part of public repo story)
docs/IntegrationGuide.md
docs/AdvocacyCaseStudy.md
docs/InterviewNarratives.md
docs/InterviewLanguagePrep.md
docs/linkedin-self-healing-vision-qa.md
86 changes: 86 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Contributing

Thanks for helping improve this project. Small, focused changes are easier to review and merge.

## Prerequisites

- Python **3.9+** (CI runs on **3.11**; matching CI locally avoids surprises).
- A clone of the repository.

## Dependencies (single source of truth)

**All pinned runtime dependencies are defined in `pyproject.toml` under `[project.dependencies]`.** Do not maintain a second copy of version pins elsewhere.

- **Developers / CI**: `pip install -e ".[dev]"` (includes pytest, coverage, Ruff, and mypy).
- **Optional**: `pip install -r requirements.txt` — this file only contains `-e .[dev]` as a convenience shim for older habits or docs that still use `-r`.

When you add or bump a dependency, edit **`pyproject.toml` only**, then reinstall your venv.

## Local setup

```bash
cd agentic_testing_framework
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
python -m pip install -U pip
pip install -e ".[dev]"
```

The CLI and tests expect `src` on the module path. Use `PYTHONPATH=src` as shown below (same as CI).

## Streamlit demo (`app.py`)

Run from the **repository root** so `agent`, `engine`, etc. resolve:

```bash
PYTHONPATH=src streamlit run app.py
```

Without `PYTHONPATH=src`, **Manual Baseline** still works; **AI Pipeline** needs the orchestrator import path above.

## Run tests

```bash
PYTHONPATH=src pytest
```

Coverage options are defined in `pyproject.toml` (`--cov=src`, XML report, and **`--cov-fail-under=34`** so total coverage cannot drift far below current levels without CI failing).

## Lint

CI runs Ruff on the full Python tree under `src` plus `tests`. Match it before opening a PR:

```bash
ruff check src tests app.py test_connection.py
```

## Type check (mypy)

Settings live in `pyproject.toml` under `[tool.mypy]`. Run the same checks as CI (two passes avoid duplicate module mapping for `src/` vs repo-root scripts):

```bash
mypy --explicit-package-bases src
MYPYPATH=src mypy --explicit-package-bases app.py test_connection.py
```

## Optional: agent smoke run (CI parity)

The workflow also runs a short end-to-end report generation:

```bash
PYTHONPATH=src python src/ai_quality_agent.py --profile dev --performance-analysis --overhead-analysis
```

## Pull requests

1. **Branch**: Open PRs against the repository default branch (usually `main`).
2. **Scope**: One logical change per PR when possible (feature, fix, or docs—not all mixed unless tightly related).
3. **Description**: Summarize *what* changed and *why*; link an issue if one exists.
4. **Green CI**: Ensure tests, Ruff, and **mypy** pass locally.
5. **Docs**: If you change CLI flags, config shape, or inference behavior, update `README.md` and any affected file under `docs/`.

## Code style

Follow existing patterns in nearby modules (logging, typing, error messages). Prefer clear names and small functions over clever one-liners.

Use **`logging.getLogger(__name__)`** instead of `print` for diagnostics. The batch CLI calls **`util.cli_logging.configure_cli_logging()`** in `__main__`, which sets `basicConfig` to include **timestamp**, **level**, and **logger name** when the root logger has no handlers yet. `app.py` does the same at import time when appropriate (e.g. under Streamlit).
24 changes: 24 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Linux container (default: amd64). llama.cpp here is a **CPU** build with OpenBLAS —
# not Apple Metal (Metal is macOS-only; use a host install if you need GPU on Apple Silicon).
FROM python:3.11-slim

RUN apt-get update && apt-get install -y \
build-essential \
cmake \
git \
libopenblas-dev \
libopencv-dev \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Dependency pins: pyproject.toml only. Runtime install (no [dev] extras).
COPY . .
RUN pip install --no-cache-dir llama-cpp-python \
&& pip install --no-cache-dir .

ENV PYTHONUNBUFFERED=1
ENV PYTHONPATH=/app/src
ENV MODEL_PATH=/app/models/your-model-q4_k_m.gguf

CMD ["python", "src/ai_quality_agent.py", "--profile", "dev"]
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Cheryl

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Loading
Loading