Lookup join drops an AND conjunct when the ON clause also has an OR over indexed columns (wrong results)
A lookup join returns rows that fail its ON condition. If the condition is <indexed col> = <const> AND (<indexed col> = r.x OR <col> = r.y), the planner builds a Concat of two index lookups: one for the constant conjunct, one for the first OR branch. The residual filter keeps only the OR, so the constant conjunct is never applied.
Reproduction
create table deps (id int primary key, type varchar(16), col_a varchar(32), col_b varchar(32), key k_type (type), key k_a (col_a));
insert into deps values (1, 'keep', 'X', null), (2, 'drop', 'X', null);
create table r (id varchar(32) primary key);
insert into r values ('X');
select d.id, d.type from r join deps d on d.type = 'keep' and (d.col_a = r.id or d.col_b = r.id) order by d.id;Expected: one row, 1, keep.
Actual:
id,type
1,keep
2,dropRow 2 has type = 'drop', so it fails d.type = 'keep'.
Plan
explain plan output for the query above; the columns: lines under the two index accesses are omitted.
Project
├─ columns: [d.id]
└─ LookupJoin
├─ ((d.col_a = r.id) OR (d.col_b = r.id))
├─ Table
│ ├─ name: r
│ └─ columns: [id]
└─ TableAlias(d)
└─ Concat
├─ TableAlias(d)
│ └─ IndexedTableAccess(deps)
│ ├─ index: [deps.type]
│ └─ keys: 'keep'
└─ TableAlias(d)
└─ IndexedTableAccess(deps)
├─ index: [deps.col_a]
└─ keys: r.idtype = 'keep' becomes one branch of the union rather than a filter applied to both branches. The join filter keeps only the OR, and row 2 matches it through col_a = r.id.
What triggers it
Same data and query, varying only the indexes:
| Secondary indexes | Rows returned |
|---|---|
type, col_a, col_b |
1, 2 (wrong) |
type, col_a |
1, 2 (wrong) |
col_a, col_b |
1 |
type only |
1 |
col_a only |
1 |
| none | 1 |
With both indexes present:
- Moving
d.type = 'keep'fromONtoWHEREgives the same wrong result. IGNORE INDEX (...)ondepsdoesn't change it.- Without the join, the same predicate is correct:
select id from deps where type = 'keep' and (col_a = 'X' or col_b = 'X')returns 1. - The same join without the OR (
on d.type = 'keep' and d.col_a = r.id) is correct.
Versions
Reproduced on dolt 2.3.1 and 2.3.5 (latest), Linux amd64, on a fresh dolt init database with the dolt sql CLI.
Impact
We found this in a schema migration that recomputes a derived "is blocked" flag with a recursive CTE joining on d.type = 'parent-child' AND ((r.kind = 'issue' AND d.depends_on_issue_id = r.id) OR (r.kind = 'wisp' AND d.depends_on_wisp_id = r.id)). The dropped type predicate let other dependency types through the join and silently mis-flagged rows. There was no error, only wrong data. The reproduction above is that pattern with the recursion and the extra branch conditions removed.
Source: dolthub/dolt