Skip to content

[codex] Emit compact SQLite IN filters - #2

Draft
wtsnz wants to merge 2 commits into
mainfrom
codex/sqlite-in-list-membership
Draft

[codex] Emit compact SQLite IN filters#2
wtsnz wants to merge 2 commits into
mainfrom
codex/sqlite-in-list-membership

Conversation

@wtsnz

@wtsnz wtsnz commented May 19, 2026

Copy link
Copy Markdown
Owner

This came out of a local SQLite app I’m building where I've got a route that asks for availability across a lot of media IDs, and the Ash filter is pretty ordinary:

filter expr(media_item_id in ^arg(:media_item_ids) and best == true)

AshSqlite currently turns that into a long OR chain:

media_item_id = ? OR media_item_id = ? OR media_item_id = ? ...

Which works for small lists, but not for long lists. SQLite has a max expression depth (1000 terms) which causes the query to crash before it can run.

So chatting to my robots, I figured that these types of queries should use SQLite’s normal scalar IN (?, ...) shape for scalar lists, while keeping the existing fallback behaviour for other complex values. This PR adjusts the generated SQL so that we use the more efficient IN query when possible.

History

It looks like the existing OR expansion was added as a fix or workaround. See this ash-project/ash_sqlite@83ce541, fix: remove list literal usage for in in ash_sqlite.

There isn’t an explanatory PR or test attached to that commit, so I’m not 100% certain for why this was implemented. My read is that AshSqlite needed to stop using the shared list-literal RHS path for SQLite, and expanding to OR was a safe local workaround because each left == value reused the existing equality/type-casting path?

This PR adjusts this path slightly so that we can use the more efficient IN shape for scalar lists.

Performance

Got my LLM to run some comparisons and the new IN approach (spoiler, its faster!) https://gist.github.com/wtsnz/5cfcf6625625719c18be9ab2b0f6d34b

Validation

  • MIX_ENV=test mix test test/filter_test.exs
  • MIX_ENV=test mix test
  • mix format --check-formatted lib/sql_implementation.ex test/filter_test.exs
  • git diff --check -- lib/sql_implementation.ex test/filter_test.exs

Contributor checklist

Leave anything that you believe does not apply unchecked.

  • I accept the AI Policy, or AI was not used in the creation of this PR.
  • Bug fixes include regression tests
  • Chores
  • Documentation changes
  • Features include unit/acceptance tests
  • Refactoring
  • Update dependencies

wtsnz added 2 commits May 19, 2026 21:53
Generate scalar SQLite IN filters as compact IN clauses instead of left-deep OR trees, while keeping the existing OR fallback for complex RHS values.

Dump RHS values through the selected SQLite adapter so booleans, datetimes, decimals, UUIDs, and custom storage types keep the behavior the old equality path provided.
Add regression coverage for large and empty pinned lists, complex RHS fallback behavior, typed scalar values, and the typed integer SQL shape used to avoid left-hand casts.
@wtsnz
wtsnz force-pushed the codex/sqlite-in-list-membership branch from 1824f70 to 37a44e1 Compare May 20, 2026 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant