CWE xxe search fixes - #22
Open
Bornunique911 wants to merge 25 commits into
Open
Conversation
…nitor PR OWASP#823 reintroduced Neo4j/Redis fallback on Heroku cache misses, causing 503s when Neo4j DNS fails. Serve precomputed GA from Postgres only on Heroku and return 404 on cache miss. Add monitor_ga_health.py for production regression alerting (especially HTTP 503). Fixes OWASP#923 Co-authored-by: Cursor <cursoragent@cursor.com>
Cloudflare blocks anonymous urllib requests to ga_standards on production. Co-authored-by: Cursor <cursoragent@cursor.com>
Allow AGENTS.md through the *.md gitignore exception and document that Heroku/opencreorg gap analysis is cache-only (no compute on production). Co-authored-by: Cursor <cursoragent@cursor.com>
Guard add_gap_analysis_result so non-material {"result":{}} primary rows
are not inserted and cannot overwrite material cache; subresource keys unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
Supports postgres-to-postgres sync via temp-table merge for prod tables without a unique index on cache_key. Co-authored-by: Cursor <cursoragent@cursor.com>
Document operational scripts and weekly prod GA checks in AGENTS.md; add link_pci_dss_cre.py for embedding-based CRE linking. Harden primary GA cache key detection, sync script materiality guards, monitor 503 test, and DSN redaction. Co-authored-by: Cursor <cursoragent@cursor.com>
Avoid accidental production writes when running link_pci_dss_cre.py without explicit --cache-file or CRE_CACHE_FILE. Co-authored-by: Cursor <cursoragent@cursor.com>
Guard against an empty get_CREs result so callers get None instead of IndexError when a DB row exists but no matching CRE document is found. Co-authored-by: Cursor <cursoragent@cursor.com>
Serve precomputed OpenCRE GA from cache on Heroku instead of computing on the web dyno, expand backfill to include automatic CRE links, and harden PCI DSS / Secure Headers imports with better linking and parser fixes. Co-authored-by: Cursor <cursoragent@cursor.com>
Harden PCI env parsing, tighten sync script safety checks, make bridge fallback tests deterministic, and format files flagged by CI black. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Track AGENTS.md and .cursor/rules so the team shares human-plan-then-agent-execute workflows, CI/PR policies, and domain safety guardrails. Co-authored-by: Cursor <cursoragent@cursor.com>
Add modular .cursor/rules for requirements gates, tickets, TDD, and verification; tighten plan-first and multi-agent flows; slim AGENTS.md to an index aligned with make lint/mypy/test checks. Co-authored-by: Cursor <cursoragent@cursor.com>
The test expected tags="1,2" but dbNodeFromCode joins the input list ["111-111", "222-222"] with commas, producing "111-111,222-222". The expected value in the test was wrong.
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Closes OWASP#862 request.args.get('text') returns None if the query param is absent. Passing None into db.text_search() causes re.search() to raise TypeError: expected string or bytes-like object. Return a 400 before reaching the database call.
Align spreadsheet_test with get_all_values-based read path so section codes like 5.10 stay strings instead of being float-coerced. Co-authored-by: Cursor <cursoragent@cursor.com>
Handle empty worksheets and pad short rows so section IDs are preserved as strings without IndexError or truncated dict keys.
Fail fast with GSpreadException when worksheet header row contains duplicates, use zip(strict=True) after row padding, and add regression tests for the helper and read_spreadsheet integration.
Fix production H12 timeouts from GET /all_cres?per_page=1000 by batching N+1 link hydration in the DB layer, capping per_page at 100, scoping DataProvider to Explorer routes with incremental page loads, and using ensureFullExplorerData for graph views. Closes OWASP#930. Related: OWASP#847, OWASP#848. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix buildTree sibling keyPath mutation, serialize loadPage via promise chain with exposed dataLoadError, hoist Explorer layout wrappers, surface load failures in graph views, restore viewport zoom, and harden pagination link parity test. Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(api): add feature-flagged /rest/v1/health endpoint
Adds a lightweight deploy/uptime health probe at GET /rest/v1/health, gated behind the CRE_ENABLE_HEALTH feature flag (off by default).
Behavior:
- Flag off (default): endpoint returns 404, as if it does not exist.
- Flag on, healthy: 200 with {ok, db_reachable, cre_count, standards_count} when the serving DB is reachable and holds a non-empty dataset.
- Flag on, unhealthy: 503 when the DB is unreachable or the dataset is empty/broken (reason explains which).
Node_collection.health_check() runs cheap COUNT queries over CRE and Node, never raises (connectivity errors are reported as ok=False), and treats a zero count for either as an empty dataset.
Scope is intentionally limited to DB reachability + data sanity. Deeper checks (gap-analysis completeness, mapping coverage, Neo4j, Redis) are deliberately excluded by design and belong in ops tooling.
* fix: load .env in feature_flags and document CRE_ENABLE_HEALTH flag
* Modified the .env issue
…ASP#939) * cwe: add XXE search aliases and enforce prohibited CWE exclusions This commit builds on the earlier CWE mapping work (PR OWASP#823) and addresses two gaps found during local verification after the previous merge. Problem: Searching for terms like 'XXE' did not reliably surface the expected top-level CWE result, even when the corresponding CWE data existed locally. This created a mismatch between the intended curated CWE import behavior and actual API/UI discoverability. Solution: - Expand text-search aliases to include common security shorthand terms (e.g., XXE), ensuring related CWE and CRE records surface correctly. - Strengthen CWE import filtering to consistently exclude prohibited entries. Scope: - CWE import filtering logic - Search alias expansion for XXE and related terms - Unit tests covering prohibited CWE exclusion and XXE search behavior - Script messaging update for the CWE refresh workflow Testing: ./venv/bin/python -m pytest application/tests/cwe_parser_test.py -k 'register_CWE or prohibited' -q ./venv/bin/python -m pytest application/tests/db_test.py -k 'text_search' -q Follows-up: OWASP#823 References: OWASP#472 * Update application/database/db.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Bornunique911 <69379200+Bornunique911@users.noreply.github.com> * Updated the code as per CodeRabbit review. * Fixed linting issues. * Added import for SQLAlchemyError and narrow exception handler in health_check. * Fixed linting issues. * Fixed whitespace-only searches issue. --------- Signed-off-by: Bornunique911 <69379200+Bornunique911@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Bornunique911
force-pushed
the
cwe-xxe-search-fixes
branch
from
July 23, 2026 17:16
cf7e21b to
912e2a7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.