feat(web_search): SearXNG meta-search with DuckDuckGo fallback - #68
Merged
Conversation
Adds an optional SearXNG backend (self-hosted meta-search, no single
upstream engine's bot detection to fight since it aggregates several)
for web_search, tried first when configured. Any failure — unreachable,
timeout, non-200, malformed JSON, JSON format disabled on the instance —
falls through to the existing DuckDuckGo HTML scraping rather than
erroring the tool call. A down or misconfigured self-hosted instance
must never make search unavailable when a zero-infra fallback already
works.
- nullain_tools.web_search: _searxng_search() queries {base_url}/search
?format=json, returns None (not an exception) on any failure signal.
create_web_search_tool(searxng_base_url=...) tries it first, falls
through to the untouched DuckDuckGo parsing path.
- nullain.config.WebFetchConfig: new searxng_base_url field ([web_fetch]
section — same conceptual policy as web_search_headers, no separate
section). None (default) preserves exact prior behavior.
- register_default_tools/Agent facade: threaded through the same way as
web_fetch_headers/web_search_headers.
10 new tests: SearXNG success skips DDG entirely, SearXNG failure modes
(unreachable, HTTP error, malformed JSON, empty results) all fall
through to DDG, and default (searxng_base_url=None) never attempts to
reach any SearXNG endpoint — identical to pre-existing behavior.
Infra note: operating a SearXNG instance (+ typically Redis/Valkey for
its own rate limiter) is the deploying operator's responsibility — this
SDK change only knows how to consume a configured instance's JSON API.
4 tasks
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional SearXNG backend for
web_search— self-hosted meta-search, aggregates multiple upstream engines rather than depending on any single one's bot-detection posture. Tried first when configured via[web_fetch].searxng_base_url(or the app-sideSEARXNG_BASE_URLenv var, once threaded through); any failure (unreachable, timeout, non-200, malformed JSON, JSON format disabled on the instance) falls through to the existing DuckDuckGo HTML scraping — a misconfigured or down self-hosted instance must never make search unavailable when a working zero-infra fallback exists.Increment 1 of 4 in the approved plan (SearXNG → Crawl4AI/Wayback → Postgres batching → OTLP+StateSummary).
Changes
nullain_tools.web_search:_searxng_search()queries{base_url}/search?format=json, returnsNone(never raises) on any failure signal.create_web_search_tool(searxng_base_url=...)tries it first, falls through to the unmodified DuckDuckGo parsing path.nullain.config.WebFetchConfig: newsearxng_base_urlfield.None(default) preserves exact prior behavior — no attempt to reach any SearXNG endpoint at all.register_default_tools/Agentfacade: threaded through the same wayweb_fetch_headers/web_search_headersalready are.Infra note (not part of this PR)
Operating a SearXNG instance (+ typically Redis/Valkey for its own rate limiter) is the deploying operator's responsibility — this change only knows how to consume a configured instance's JSON API. On Coolify: add SearXNG as a separate Resource on the same internal network, point
SEARXNG_BASE_URLat it.Test plan
searxng_base_url=None) never attempts to reach any SearXNG endpoint.test_web_search.pytests pass unmodified — confirms the DDG path itself wasn't touched by the refactor.uv run ruff check ./uv run ruff format --check ./uv run pyright— all clean (same invocations CI uses).