feat: implement dynamic analyzer discovery and risk score validation#74
feat: implement dynamic analyzer discovery and risk score validation#74umran666 wants to merge 3 commits into
Conversation
Resolves TODO A.2.1-A.2.4 (Analyzer discovery) and TODO A.3.2 (Risk score assertion). Signed-off-by: umran666 <shaikumran666@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
The move to dynamic analyzer discovery is reasonable and removes a lot of boilerplate, and wiring requires_api_key / is_available() gating into create_graph matches the intent of the TODO it resolves. A few issues, one of which I'd consider blocking for a security tool.
1. Import failures are silently swallowed (blocking). In _discover_analyzers, a module that fails to import is caught with except Exception and logged at logger.debug(...), then skipped. Since the default log level is WARNING, a broken analyzer (syntax error, bad import, refactor mistake) silently disappears from every scan with no visible signal — the scan still completes and can report a low/zero risk while an entire detection category isn't running. That's a meaningful regression from the previous explicit-import behaviour, which failed loudly. Please log this at WARNING/ERROR (and ideally surface it in output) so a dropped analyzer is never invisible. It would also help to distinguish "module isn't an analyzer" (no ANALYZER_ID) from "an analyzer failed to import".
2. Confirm the gating attributes actually exist. The new skip logic reads getattr(mod, "is_available", None) and getattr(mod, "requires_api_key", False). The static analyzer modules don't define these (correctly — they need no key), but if the goal is to skip the LLM/semantic analyzers when no key is present, those modules must declare requires_api_key = True; otherwise they default to False and are always wired, making the gating a no-op. Please confirm the LLM-dependent analyzers actually set this.
3. Edge case: if discovery/gating ends up skipping every analyzer, wired_analyzers is empty and nothing connects to meta_analyzer. Worth a guard or at least a warning.
Non-blocking nits:
- Discovery order now depends on
pkgutil.iter_modules(filesystem order) rather than the curated list, and the registry test was loosened to a set comparison. Fine if analyzer order is genuinely irrelevant (parallel fan-out), but a comment noting that ordering is no longer guaranteed would help future readers. - Several of the new lines in
graph.pycarry trailing whitespace, which will likely tripruff/lint.
Happy to re-review once import failures are visible and the requires_api_key wiring is confirmed.
Signed-off-by: umran666 <shaikumran666@gmail.com>
|
Thanks for the review. I've pushed a commit addressing all the feedback:
|
Signed-off-by: umran666 <shaikumran666@gmail.com>
| from skillspector.nodes.resolve_input import resolve_input | ||
| from skillspector.state import SkillspectorState | ||
|
|
||
| logger = get_logger(__name__) |
There was a problem hiding this comment.
[Automated SkillSpector Review]
Non-blocking (formatting): this module-level assignment leaves only one blank line before def create_graph(). ruff format (Black style) enforces two blank lines before a top-level def, so ruff format --check will flag this file — please run make format. Same pattern before def _discover_analyzers() in nodes/analyzers/__init__.py.
rng1995
left a comment
There was a problem hiding this comment.
[Automated SkillSpector Review]
Re-review: blockers resolved — approving.
- Import failures are no longer silent:
_discover_analyzerscatchesImportError/Exceptionand logs atERROR(distinguishing import failure from "not an analyzer"). requires_api_key = Trueis now declared on the three semantic analyzers andcreate_graphgates them onis_llm_available(), so the skip logic is a real gate rather than a no-op.- An empty-
wired_analyzerswarning guard was added.
Non-blocking: the inserted module-level statements leave a single blank line before def create_graph() (in graph.py) and before def _discover_analyzers() (in __init__.py); ruff format wants two, so please run make format to avoid a CI format-check failure.
rng1995
left a comment
There was a problem hiding this comment.
Re-review — approving. The only change since my last approval is a style: run ruff format commit (graph.py, analyzers/__init__.py); the dynamic-analyzer-discovery logic I previously approved is unchanged.
Heads-up (non-blocking for the review, but blocks merge): GitHub currently reports this branch as conflicting with main (mergeable_state: dirty) — please rebase/resolve before merging.
|
Hey @rng1995 just checking in. All review feedback has been addressed and the latest push includes the ruff format fix. Is this good to merge, or is there anything else you'd like me to change? |
Resolves TODO A.2.1-A.2.4 (Analyzer discovery) and TODO A.3.2 (Risk score assertion).