design update for ragas removal and evaluation updates

[lakindu-yl]

Description

• Removal of Ragas
• Adding Deepeval
• Adding capability for user to add his/her own evaluator
• Adding capability for the user to see the metric results

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Test update
• CI/CD update
• Other (please describe):

Testing

• Unit tests pass locally
• Integration tests pass locally
• Manual testing completed
• New tests added for changes

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

Screenshots (if applicable)

Additional Notes

[Copilot]

Pull request overview

Adds a design/spec package for #555 to remove the current RAGAS-based evaluation path and move test evaluation behind a pluggable AKEvaluator interface (with DeepEval as the first built-in), including research notes on evaluator-framework input/output shapes and metric catalogues.

Changes:

• Adds a detailed design document for replacing the current RAGAS-based harness logic with a pluggable evaluator architecture and new return_metrics behavior.
• Adds a research survey comparing DeepEval/Opik/Braintrust/RAGAS to justify the evaluator payload/result envelope and mode naming.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
docs/specs/555-pluggable-test-evaluators/design.md	Design spec for pluggable evaluators, mode rename, and return_metrics result-return capability.
docs/specs/555-pluggable-test-evaluators/research/evaluator-framework-survey.md	Research survey informing the payload/result shape and portability constraints across evaluator frameworks.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[amithad]

Spec-only PR review (design.md + research/ for #555). Reviewed against ak-dev-write-spec, ak-dev-architecture, ak-dev-testing-conventions, and ak-dev-code-quality; no implementation code, so spec-conformance review of code was skipped.

Overall: a strong, unusually well-evidenced design at the right stage (design.md + research shipped together, no spec.md skipping ahead). Nearly every path:line citation verified exactly against develop (test.py, config.py, pyproject.toml, factory.py, guardrail/sandbox factories, chat_service, skills, docs, tests, example call sites), and the research survey's AK-code claims spot-checked clean. Behavioural changes are called out explicitly, non-goals are thorough, and unresolved items are honestly parked in "Verification required in spec.md". The one substantive problem is the Migration surface section: its enumerations do not survive verification, and following it as written ships broken examples.

Findings (inline): 1 blocker, 5 suggestions, 1 question, 2 nits.

• [blocker] Explicit-threshold call sites: 14 example test files carry them (30 sites, values 10/20/80, plus one match_threshold=20), not just nats/kafka; all raise ValueError on the new 0-1 scale.
• [suggestion] "every one of them carrying a judge: block" is false for the 6 sandbox mode: fuzzy configs.
• [suggestion] Missing surfaces: use-cases/waste-sorting-assistant/test-config.yaml (hard-fails the new mode pattern), ak-py/README.md:967, .agents/skills/ak-dev-testing-conventions/SKILL.md:269-275, ak-py/src/agentkernel/skills/ak-test/evals/evals.json:33-48.
• [suggestion] Motivation cites an untracked file no reviewer can verify.
• [question] fallback x AKMetricNotSupported interaction is unspecified.
• [suggestion] DeepEval's async_mode default conflicts with the running-event-loop constraint the design itself establishes.
• [suggestion] The evaluator-lifetime block (and parts of the DeepEval sections) is spec.md-altitude detail in a design.md.
• [nit]s: one citation drift (test.py:183 vs :188); pytest-asyncio is already in the test extra.

Not anchorable to the diff (PR metadata, per ak-dev-write-spec PR guidance):

• Spec-only PRs should carry a docs: conventional-commit title (e.g. docs: add design spec for #555 pluggable test evaluators); the current title has no type prefix.
• The template marks "New feature" alongside "Documentation update" and leaves the Testing/Checklist sections untouched; for a spec-only PR, mark Documentation update, state that items like tests do not apply, and say explicitly that implementation follows in a separate PR.

No existing PR feedback, so nothing was deduplicated. CI is fully green.