Skip to content

Make KNN filters keep the rows SQLite keeps - #329

Open
MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:constraint-affinity
Open

MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:constraint-affinity

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 7, 2026 •

Copy link
Copy Markdown

vec0 checks a KNN query's constraints itself and sets aConstraintUsage[].omit, which tells SQLite a constraint "is assumed to be fully handled by the virtual table and might not be checked again" (sqlite3.h). Each constraint vec0 claims therefore has to keep exactly the rows SQLite would keep for the same WHERE clause, and today the same WHERE clause keeps different rows with and without a match:

create virtual table v using vec0(a float[1], n integer, chunk_size=8);
insert into v(rowid, a, n) values (1, '[1]', 0), (2, '[2]', 5);

select rowid from v where n = 'abc';                               -- no rows
select rowid from v where a match '[0]' and k = 10 and n = 'abc';  -- 1

On main:

  • n = ? bound to NULL or 'abc' keeps n = 0, n = 5.5 keeps n = 5, n < 5.5 drops n = 5, and n in (5.5, 6) keeps n = 5.
  • f > 'abc' keeps every positive f, though every number is less than any text.
  • b = 2 keeps the true rows.
  • t in (null) keeps t = '', and t = x'78' keeps t = 'x'.
  • t < ? bound to NULL crashes (a NULL pointer read) once a chunk holds a text longer than 12 bytes, and <, <=, > and >= fail with "Could not filter metadata fields" once a chunk holds a text of exactly 12 bytes.
  • Text comparisons ignore the query's collation: t = 'abc' COLLATE NOCASE misses 'ABC', and COLLATE RTRIM misses 'abc '.
  • A partition key compares as stored: p = '5' misses p = 5, s = 5 misses s = '5', and collations are ignored.
  • rowid in (5.5, 'abc', null) matches rowids 5 and 0, and a point lookup rowid = ? matches rowid 0 for NULL, 'abc' and a blob, and rowid 5 for 5.5. With a text primary key, an id in id in (...) that names no row empties the whole result.
  • distance > ? bound to NULL keeps every row and distance < 'abc' none, and distances compared in f32 (the TODO beside that cast) keep the row at distance 2 for distance <= 1.999999999999, which SQLite drops.

This follows SQLite's R-Tree, the extension that checks its own constraints:

  • vec0 declares its columns with their types, as rtreeInit() does: metadata columns boolean, integer, float or text, partition keys and a text primary key with theirs, rowid integer and distance real. SQLite's own checks then give a value the affinity vec0 gives it.
  • Each filter gives the constraint's value its column's affinity and compares by Datatypes In SQLite, 4.1 and 4.2: NULL compares with nothing, INTEGER and REAL compare exactly, as sqlite3IntFloatCompare() does, every number sorts before any text, and every text before any blob. A rowid lookup takes the value as rtreeFilter() does: numeric affinity, then an integer or a real equal to one.
  • Text compares under the constraint's collation, from sqlite3_vtab_collation(): BINARY, NOCASE and RTRIM as binCollFunc, nocaseCollatingFunc and rtrimCollFunc do, and any other collation is refused with an error. SQLite names no collation for != (it hands it over without its left operand), so vec0 keeps the rows unequal under BINARY and leaves != for SQLite to check again. SQLite orders text under BINARY in the database's encoding, so in a UTF-16 database vec0 leaves <, <=, > and >= on text to SQLite.
  • A distance compares as the double SQLite receives.

The tests compare each filter with a plain table declaring the same types and with SQLite's own scan of the vtab: every operator over NULL, integers, reals, numeric and other text and blobs, texts around the 12 bytes a chunk keeps inline, each collation, in (...) lists, partition keys, rowid and text id lookups, distances a hair either side of a row's, and UTF-16 databases. A KNN query whose k is the number of matching rows must return all of them, so a filter that keeps any other row fails. The tests fail on main and pass here. The full suite passes (245 passed, 66 skipped, 204 snapshots), also built with AddressSanitizer and UBSan against a SQLite built with AddressSanitizer. make test-unit does not build on main (tests/sqlite-vec-internal.h redeclares the rescore types), and this change adds no error to it.

The same change is open on vlasky's fork as vlasky#12, which also covers its LIKE, GLOB, IS and IS NOT filters.

vec0 checks a KNN query's constraints itself and sets
aConstraintUsage[].omit, which tells SQLite a constraint "is assumed to
be fully handled by the virtual table and might not be checked again"
(sqlite3.h). Each constraint vec0 claims therefore has to keep exactly
the rows SQLite would keep for the same WHERE clause. They did not:

- `n = ?` bound to NULL or 'abc' kept n = 0, `n = 5.5` kept n = 5,
  `n < 5.5` dropped n = 5, and `n in (5.5, 6)` kept n = 5.
- `f > 'abc'` kept every positive f, though every number is less than
  any text.
- `b = 2` kept the true rows.
- `t in (null)` kept t = '', and `t = x'78'` kept t = 'x'.
- `t < ?` bound to NULL crashed (a NULL pointer read) once a chunk held
  a text longer than 12 bytes, and <, <=, > and >= failed with "Could
  not filter metadata fields" once a chunk held a text of exactly 12
  bytes.
- Text comparisons ignored the query's collation: `t = 'abc' COLLATE
  NOCASE` missed 'ABC', and `COLLATE RTRIM` missed 'abc  '.
- A partition key compared as stored: `p = '5'` missed p = 5, `s = 5`
  missed s = '5', and collations were ignored.
- `rowid in (5.5, 'abc', null)` matched rowids 5 and 0, and a point
  lookup `rowid = ?` matched rowid 0 for NULL, 'abc' and a blob, and
  rowid 5 for 5.5. With a text primary key, an id in `id in (...)` that
  names no row emptied the whole result.
- `distance > ?` bound to NULL kept every row and `distance < 'abc'`
  none, and distances compared in f32 (the TODO beside that cast) kept
  the row at distance 2 for `distance <= 1.999999999999`, which SQLite
  drops.

vec0 now follows SQLite's R-Tree, the extension that checks its own
constraints:

- It declares its columns with their types, as rtreeInit() does:
  metadata columns boolean, integer, float or text, partition keys and
  a text primary key with theirs, rowid integer and distance real. So
  SQLite's own checks give a value the affinity vec0 gives it.
- Each filter gives the constraint's value its column's affinity and
  compares by https://www.sqlite.org/datatype3.html#comparison_expressions:
  NULL compares with nothing, INTEGER and REAL compare exactly, as
  sqlite3IntFloatCompare() does, every number sorts before any text,
  and every text before any blob. A rowid lookup takes the value as
  rtreeFilter() does: numeric affinity, then an integer or a real
  equal to one.
- Text compares under the constraint's collation, from
  sqlite3_vtab_collation(): BINARY, NOCASE and RTRIM as binCollFunc,
  nocaseCollatingFunc and rtrimCollFunc do, and any other collation is
  refused with an error. SQLite names no collation for !=, so vec0
  keeps the rows unequal under BINARY and leaves != for SQLite to check
  again. SQLite orders text under BINARY in the database's encoding,
  so in a UTF-16 database vec0 leaves <, <=, > and >= on text to
  SQLite.
- A distance compares as the double SQLite receives.

The tests compare each filter with a plain table declaring the same
types and with SQLite's own scan of the vtab: every operator over NULL,
integers, reals, numeric and other text and blobs, texts around the 12
bytes a chunk keeps inline, each collation, IN lists, partition keys,
rowid and text id lookups, distances a hair either side of a row's,
and UTF-16 databases. A KNN query whose k is the number of matching
rows must return all of them, so a filter that keeps any other row
fails. The tests fail on main and pass here. The full suite passes (245
passed, 66 skipped, 204 snapshots), also built with AddressSanitizer
and UBSan against a SQLite built with AddressSanitizer. make test-unit
does not build on main (tests/sqlite-vec-internal.h redeclares the
rescore types), and this change adds no error to it.
@MayCXC
MayCXC force-pushed the constraint-affinity branch from 38c59a5 to 2455320 Compare October 7, 2026 18:16
@MayCXC MayCXC changed the title Compare KNN filter values the way SQLite compares them Make KNN filters keep the rows SQLite keeps Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant