Behavioral scoring pipeline for anonymized AI conversations.
This repository owns the MVP scoring layer that turns anonymized conversations into research-friendly indicators. The first version should be transparent, auditable, and easy to compare against expert labels before any automated claims are trusted.
- Consume anonymized conversation records.
- Score conversations across the Make AI Visible behavioral dimensions.
- Return structured scores, confidence, evidence snippets, and model/rubric version metadata.
- Support both transparent rubric baselines and future model-backed scoring.
- Provide an evaluation harness for agreement with human reviewers.
This engine should never receive raw, non-anonymized submissions. Tests and examples must use synthetic or already anonymized content.
- Python package with a small CLI.
- Pydantic schemas for scoring input/output.
- Pytest evaluation fixtures.
- Baseline rubric implementation before LLM/model integration.
Implement a deterministic baseline scorer with fixtures, schema validation, and a report comparing output against hand-labeled examples.
This repo includes a deterministic keyword baseline for synthetic, anonymized records. It is intentionally transparent and marked as unvalidated for research claims.
Run it locally:
python -m scoring_engine.cli examples/anonymized_conversation.jsonThe baseline outputs dimension scores, confidence values, evidence keywords, and a rubric version. It must only receive anonymized records.