Skip to content

Emit SEMI join inner conditions in the ON clause - #347

Draft
JoshDreamland wants to merge 1 commit into
mainfrom
fix-semi-join-inner-conds
Draft

Emit SEMI join inner conditions in the ON clause#347
JoshDreamland wants to merge 1 commit into
mainfrom
fix-semi-join-inner-conds

Conversation

@JoshDreamland

@JoshDreamland JoshDreamland commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

A SEMI or ANTI join uses its ON clause as the match test, so every condition that references the inner relation must appear there. ClickHouse's LEFT SEMI JOIN exposes one arbitrary matching row to the WHERE clause. Testing an inner condition against that row can miss another qualifying match and return too few rows.

Do not rely on PostgreSQL's is_pushed_down flag to place these conditions. PostgreSQL sets it for semijoin conditions because their placement does not affect its executor, but the distinction matters to
a remote ClickHouse join. Move every condition that references the inner relation into the remote ON clause.

ClickHouse versions before 26.3 reject non-equality ON conditions that span both sides when join_use_nulls is enabled; 23.x rejects them regardless of that setting. Keep affected joins local on those servers. Resolve the user mapping as the executor would, and disable pushdown if no mapping is available.

Add EXISTS and NOT EXISTS tests that verify results whether the join runs remotely or locally.

Claude Summary

An EXISTS whose correlation condition references both sides was deparsed into the WHERE clause instead of the join's ON clause, silently returning too few rows. ClickHouse's LEFT SEMI JOIN surfaces the column values of one arbitrary matching row, so the condition interrogated that row rather than asking whether any matching row satisfied it.

SELECT count(*) FROM orders o WHERE EXISTS (
  SELECT 1 FROM lineitem l WHERE l.l_orderkey = o.o_orderkey AND l.l_suppkey <> o.o_custkey);
pushed down local
above query (TPC-H SF1) 1499990 1499998
TPC-H Q21, EXISTS-only shape 56824 73089

No NULLs are involved — every column above is NOT NULL, verified via attnotnull. It is purely the scope of the existential quantifier, demonstrable in ClickHouse with no nullable types at all:

-- right side has two matches for k=1: w=10 and w=20
LEFT SEMI JOIN ... ON l.k = r.k              -- surfaces w = 10 (one arbitrary match)
  ... ON l.k = r.k          WHERE r.w <> 10  -- 0 rows  <- wrong
  ... ON l.k = r.k AND r.w <> 10             -- 1 row   <- correct

@JoshDreamland
JoshDreamland force-pushed the fix-semi-join-inner-conds branch 2 times, most recently from 60b40d1 to 068b462 Compare August 14, 2026 17:16
@JoshDreamland
JoshDreamland marked this pull request as draft August 14, 2026 17:20
@JoshDreamland
JoshDreamland force-pushed the fix-semi-join-inner-conds branch from 068b462 to a03e55b Compare August 17, 2026 15:55
A SEMI or ANTI join uses its ON clause as the match test, so every
condition that references the inner relation must appear there.
ClickHouse's LEFT SEMI JOIN exposes one arbitrary matching row to the
WHERE clause. Testing an inner condition against that row can miss
another qualifying match and return too few rows.

Do not rely on PostgreSQL's `is_pushed_down` flag to place these
conditions. PostgreSQL sets it for semijoin conditions because their
placement does not affect its executor, but the distinction matters to
a remote ClickHouse join. Move every condition that references the
inner relation into the remote ON clause.

ClickHouse versions before 26.3 reject non-equality ON conditions that
span both sides when join_use_nulls is enabled; 23.x rejects them
regardless of that setting. Keep affected joins local on those servers.
Resolve the user mapping as the executor would, and disable pushdown if
no mapping is available.

Add `EXISTS` and `NOT EXISTS` tests that verify results whether the join
runs remotely or locally.
@JoshDreamland
JoshDreamland force-pushed the fix-semi-join-inner-conds branch from a03e55b to f49ddb0 Compare August 17, 2026 20:55
@theory
theory requested a review from serprex August 18, 2026 16:55
@theory theory added pushdown Improvements to query pushdown sql Improve SQL coverage or FDW capabilities labels Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pushdown Improvements to query pushdown sql Improve SQL coverage or FDW capabilities

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants