Skip to content

feat(web_search): SearXNG meta-search with DuckDuckGo fallback - #68

Merged
netty-linux merged 1 commit into
masterfrom
feat/searxng-web-search
Aug 9, 2026
Merged

feat(web_search): SearXNG meta-search with DuckDuckGo fallback#68
netty-linux merged 1 commit into
masterfrom
feat/searxng-web-search

Conversation

@netty-linux

Copy link
Copy Markdown
Owner

Summary

Adds an optional SearXNG backend for web_search — self-hosted meta-search, aggregates multiple upstream engines rather than depending on any single one's bot-detection posture. Tried first when configured via [web_fetch].searxng_base_url (or the app-side SEARXNG_BASE_URL env var, once threaded through); any failure (unreachable, timeout, non-200, malformed JSON, JSON format disabled on the instance) falls through to the existing DuckDuckGo HTML scraping — a misconfigured or down self-hosted instance must never make search unavailable when a working zero-infra fallback exists.

Increment 1 of 4 in the approved plan (SearXNG → Crawl4AI/Wayback → Postgres batching → OTLP+StateSummary).

Changes

  • nullain_tools.web_search: _searxng_search() queries {base_url}/search?format=json, returns None (never raises) on any failure signal. create_web_search_tool(searxng_base_url=...) tries it first, falls through to the unmodified DuckDuckGo parsing path.
  • nullain.config.WebFetchConfig: new searxng_base_url field. None (default) preserves exact prior behavior — no attempt to reach any SearXNG endpoint at all.
  • register_default_tools/Agent facade: threaded through the same way web_fetch_headers/web_search_headers already are.

Infra note (not part of this PR)

Operating a SearXNG instance (+ typically Redis/Valkey for its own rate limiter) is the deploying operator's responsibility — this change only knows how to consume a configured instance's JSON API. On Coolify: add SearXNG as a separate Resource on the same internal network, point SEARXNG_BASE_URL at it.

Test plan

  • 10 new tests: SearXNG success skips DDG entirely; SearXNG failure modes (unreachable, HTTP error, malformed JSON, empty results) all fall through to DDG; default (searxng_base_url=None) never attempts to reach any SearXNG endpoint.
  • All 10 pre-existing test_web_search.py tests pass unmodified — confirms the DDG path itself wasn't touched by the refactor.
  • Full suite: 658 passed, 7 skipped, 84.75% coverage.
  • uv run ruff check . / uv run ruff format --check . / uv run pyright — all clean (same invocations CI uses).

Adds an optional SearXNG backend (self-hosted meta-search, no single
upstream engine's bot detection to fight since it aggregates several)
for web_search, tried first when configured. Any failure — unreachable,
timeout, non-200, malformed JSON, JSON format disabled on the instance —
falls through to the existing DuckDuckGo HTML scraping rather than
erroring the tool call. A down or misconfigured self-hosted instance
must never make search unavailable when a zero-infra fallback already
works.

- nullain_tools.web_search: _searxng_search() queries {base_url}/search
  ?format=json, returns None (not an exception) on any failure signal.
  create_web_search_tool(searxng_base_url=...) tries it first, falls
  through to the untouched DuckDuckGo parsing path.
- nullain.config.WebFetchConfig: new searxng_base_url field ([web_fetch]
  section — same conceptual policy as web_search_headers, no separate
  section). None (default) preserves exact prior behavior.
- register_default_tools/Agent facade: threaded through the same way as
  web_fetch_headers/web_search_headers.

10 new tests: SearXNG success skips DDG entirely, SearXNG failure modes
(unreachable, HTTP error, malformed JSON, empty results) all fall
through to DDG, and default (searxng_base_url=None) never attempts to
reach any SearXNG endpoint — identical to pre-existing behavior.

Infra note: operating a SearXNG instance (+ typically Redis/Valkey for
its own rate limiter) is the deploying operator's responsibility — this
SDK change only knows how to consume a configured instance's JSON API.
@netty-linux
netty-linux merged commit c3c281e into master Aug 9, 2026
11 checks passed
@netty-linux
netty-linux deleted the feat/searxng-web-search branch August 9, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant