Skip to content

Add repository-owned deterministic validation before PR creation #174

Description

@eshulman2

Problem

Forge can create a pull request before running repository-specific deterministic checks such as lint, unit tests, type checks, generated-file verification, or builds. Provider CI eventually catches failures, but only after pushing and opening the PR, increasing feedback latency and consuming CI resources.

Projects need an opt-in pre-PR validation gate that developers can reproduce locally and that Forge can use for bounded automated repair.

Proposed model

Validation definitions belong to each repository. A declarative manifest references normal executable scripts checked into that repository; it must not embed shell commands or generate scripts dynamically.

Example:

version: 1
checks:
  - name: lint
    script: scripts/validation/lint.sh
    required: true
    auto_fix: true
    timeout_seconds: 300
    fix_guidance: Fix formatting and static-analysis failures without changing behavior.
  - name: generated-files
    script: scripts/validation/generated-files.sh
    required: false
    auto_fix: false
    timeout_seconds: 300

Developers must be able to run the referenced scripts directly without Forge. The manifest describes execution and policy; scripts contain executable logic.

Trust boundary

Forge must load the active validation manifest and scripts from the trusted base revision, not blindly from agent-modified workspace content. A change must not be able to weaken or replace the gate evaluating that same change.

Updates to validation policy become active only after they are reviewed and merged into the configured base branch. Execution remains sandboxed with bounded output, time, and resources.

Workflow behavior

For each repository work item, after implementation/local review and before push or PR creation:

  1. Resolve that repository's trusted validation manifest.
  2. Execute checks deterministically and capture structured results.
  3. If an auto-fix-enabled check fails, provide its bounded output and fix guidance to a dedicated repair step.
  4. Rerun the exact same trusted checks after repair.
  5. Stop after the configured repair budget.
  6. Block on exhausted required failures; record advisory failures without blocking.
  7. Include results in workflow state, Jira status, tracing, and the eventual PR description.

Validation is per repository. A multi-repository workflow may use different scripts and policy for each repository, and aggregate readiness must reflect required validation failures independently.

Relationship to existing stages

  • This gate does not replace local qualitative review.
  • It does not replace provider CI; it provides faster deterministic feedback before creating a PR/MR.
  • Agent repair consumes structured validation results rather than inferring success from prose.
  • Missing, malformed, unsafe, or non-executable required validation configuration fails closed with an actionable error.

Acceptance criteria

  • A repository can opt in with a versioned declarative manifest referencing executable scripts.
  • The manifest contains no embedded shell commands or generated-script mechanism.
  • The same scripts can be run directly by developers locally.
  • Forge evaluates policy/scripts from the trusted base revision.
  • Checks run per repository with independent output, timeout, required/advisory state, and repair eligibility.
  • Auto-fix uses a bounded validate → repair → revalidate loop.
  • Exhausted required failures prevent PR/MR creation; advisory failures are reported without blocking.
  • Missing or malformed required configuration cannot silently pass.
  • Validation results are structured, persisted, traced, and surfaced in Jira and the PR/MR summary.
  • Tests cover required/advisory checks, repair success/exhaustion, timeout, malformed configuration, attempted policy modification, and multiple repositories with different policies.

Context

Converted from proposal PR #19 after review. This issue incorporates the review direction to keep configuration declarative, use directly runnable repository scripts, and scope validation per repository rather than only by Jira-project skill resolution.

Related architectural work: #150 (typed artifact contracts) and #162 (per-repository provider/work-item model).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:executionAgent execution, containers, sandboxes, and runtime driversarea:workflowWorkflow graphs, orchestration, state, routing, and gatesciContinuous integration, checks, and CI repair flowsenhancementNew feature or requestmajorLarge or cross-cutting effort requiring coordinated design and implementation

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions