Create pipeline for symbol reachability

[ziadhany]

Issues

• cravex2-reachability: Create pipeline for symbol reachability #1756

Changes

Add Pipeline(s) that can retrieve vulnerable / fix symbols, collect local symbols (pur2sym) and match them
Pipeline Graph:

flowchart TD
    %% Main Entry Point
    Start([collect_and_store_symbol_reachability_results]) --> GetResources[Filter candidate_resources]
    GetResources --> GroupPatches[Group fixed_in_patches by vcs_url]
    GroupPatches --> RepoLoop{For each repository}

    %% Repository Processing
    RepoLoop -->|Process Repo| GitContext[GitRepositoryContext]
    GitContext -->|Clone & Enter| PatchLoop{For each patch}
    
    %% Patch Processing
    PatchLoop -->|Next Patch| GitCheckout[Checkout commit_hash]
    GitCheckout --> GenReport[generate_reachability_report]
    
    %% Patch Analysis Stage
    GenReport --> PatchAnalyzer[PatchAnalyzer.collect_patch_symbols]
    PatchAnalyzer --> AnalyzeDiff[Get Diff & Changed Files]
    AnalyzeDiff --> PatchAnalyzeLoop[Analyze vulnerable & fixed texts]
    PatchAnalyzeLoop --> DetectLang[detect_language_with_scancode]
    DetectLang --> ParsePatchAST[LanguageQuery.parse_code_to_ast]
    ParsePatchAST --> ExtractPatchSymbols[SymbolExtractor: Extract Changed Symbols]
    ExtractPatchSymbols --> DiffSymbols[diff_changed_symbols]
    DiffSymbols --> ResourceLoop{For each candidate_resource}

    %% Resource Matching Stage
    ResourceLoop -->|Next Resource| LangCheck{Language matches patch?}
    LangCheck -->|No| ResourceLoop
    LangCheck -->|Yes| ResAnalyzer[ResourceAnalyzer.build_index]
    
    ResAnalyzer --> ParseResAST[Parse Resource to AST]
    ParseResAST --> IndexResSymbols[Extract definitions, imports, calls]
    IndexResSymbols --> Matcher[ResourcePatchMatcher.match]
    
    Matcher --> MatchVuln[Match Vulnerable Symbols]
    Matcher --> MatchFixed[Match Fixed Symbols]
    
    MatchVuln & MatchFixed --> CheckEvidence{Evidence found?}
    CheckEvidence -->|No| ResourceLoop
    
    %% Reporting Stage
    CheckEvidence -->|Yes| Classify[classify_reachability]
    Classify --> StatusReach[REACHABLE / POTENTIALLY / NOT_REACHABLE]
    StatusReach --> UpdateData[Update resource.extra_data]
    UpdateData --> ResourceLoop

    %% Loop Backs and Exit
    ResourceLoop -->|All resources checked| PatchLoop
    PatchLoop -->|All patches checked| RepoLoop
    RepoLoop -->|All repos checked| End([End Process])

    %% Styling
    classDef functionNode fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px;
    classDef logicNode fill:#fff3e0,stroke:#ff9800,stroke-width:2px;
    classDef loopNode fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px;
    
    class Start,GenReport,PatchAnalyzer,ResAnalyzer,Matcher,Classify functionNode;
    class RepoLoop,PatchLoop,ResourceLoop loopNode;
    class LangCheck,CheckEvidence logicNode;

Loading

Checklist

• I have read the contributing guidelines
• I have linked an existing issue above
• I have added unit tests covering the new code
• I have reviewed and understood every line of this PR

[ziadhany]

The pipeline requires the unidiff dependency to parse diffs/patch text

[ziadhany]

This depends on:

• Migrate VulnerableCode integration to API v3 #2148

We have an API support for patching VulnerableCode, see:

• fedcode-next: Support UI and API for patches and diffs vulnerablecode#2152

[ziadhany]

I'm also not sure why Run unit tests on macOS / run-unit-tests (3.13) is failing.

[TG1999]

@ziadhany try to fix the failing test please

[TG1999]

You should only go for reachability if collect_symbols_reachability has called it, not for every VCIO call

[ziadhany]

We're currently relying on the find_vulnerabilities pipeline to retrieve vulnerabilities. One possible solution is to introduce an optional field reachability in the find_vulnerabilities pipeline.

[TG1999]

@ziadhany this mostly looks good, let's have a session soon and do a demo and understand the limitations of the current approach

[tdruez]

Is this absolutely needed? No built-in solutions available?

[ziadhany]

The unidiff library doesn't have any dependencies, and we only need a single file from it: patch.py.
I'll include it in this PR files and add .ABOUT and .LICENSE files for it.

[tdruez]

Sorry, I phrased my question badly. I wasn't asking whether unidiff is dependency-free, I was asking whether we need to parse a unified diff in the first place.

We already have both file contents from the git trees in get_changed_files(), so difflib can give us the changed line numbers directly:

def compute_changed_lines(vulnerable_text, fixed_text):
    """Return the removed and added line numbers between two file contents."""
    matcher = difflib.SequenceMatcher(
        a=vulnerable_text.splitlines(),
        b=fixed_text.splitlines(),
        autojunk=False,  # otherwise difflib ignores lines that repeat often
    )

    removed_lines = []
    added_lines = []
    for tag, vuln_start, vuln_end, fixed_start, fixed_end in matcher.get_opcodes():
        if tag == "equal":
            continue
        # opcodes are 0-based and end-exclusive, line numbers are 1-based
        removed_lines.extend(range(vuln_start + 1, vuln_end + 1))
        added_lines.extend(range(fixed_start + 1, fixed_end + 1))

    return removed_lines, added_lines

In collect_patch_symbols() this would replace a single line:

removed_lines, added_lines = compute_changed_lines(vulnerable_text, fixed_text)

Would that work as a full replacement for unidiff? I'd rather avoid a vendored copy.

[ziadhany]

You are right, we can use difflib.SequenceMatcher. I didn't notice there was a simpler solution.