Repository navigation
Conversation
vec0 checks a KNN query's constraints itself and sets aConstraintUsage[].omit, which tells SQLite a constraint "is assumed to be fully handled by the virtual table and might not be checked again" (sqlite3.h). Each constraint vec0 claims therefore has to keep exactly the rows SQLite would keep for the same WHERE clause. They did not: - `n = ?` bound to NULL or 'abc' kept n = 0, `n = 5.5` kept n = 5, `n < 5.5` dropped n = 5, and `n in (5.5, 6)` kept n = 5. - `f > 'abc'` kept every positive f, though every number is less than any text. - `b = 2` kept the true rows. - `t in (null)` kept t = '', and `t = x'78'` kept t = 'x'. - `t < ?` bound to NULL crashed (a NULL pointer read) once a chunk held a text longer than 12 bytes, and <, <=, > and >= failed with "Could not filter metadata fields" once a chunk held a text of exactly 12 bytes. - Text comparisons ignored the query's collation: `t = 'abc' COLLATE NOCASE` missed 'ABC', and `COLLATE RTRIM` missed 'abc '. - A partition key compared as stored: `p = '5'` missed p = 5, `s = 5` missed s = '5', and collations were ignored. - `rowid in (5.5, 'abc', null)` matched rowids 5 and 0, and a point lookup `rowid = ?` matched rowid 0 for NULL, 'abc' and a blob, and rowid 5 for 5.5. With a text primary key, an id in `id in (...)` that names no row emptied the whole result. - `distance > ?` bound to NULL kept every row and `distance < 'abc'` none, and distances compared in f32 (the TODO beside that cast) kept the row at distance 2 for `distance <= 1.999999999999`, which SQLite drops. vec0 now follows SQLite's R-Tree, the extension that checks its own constraints: - It declares its columns with their types, as rtreeInit() does: metadata columns boolean, integer, float or text, partition keys and a text primary key with theirs, rowid integer and distance real. So SQLite's own checks give a value the affinity vec0 gives it. - Each filter gives the constraint's value its column's affinity and compares by https://www.sqlite.org/datatype3.html#comparison_expressions: NULL compares with nothing, INTEGER and REAL compare exactly, as sqlite3IntFloatCompare() does, every number sorts before any text, and every text before any blob. A rowid lookup takes the value as rtreeFilter() does: numeric affinity, then an integer or a real equal to one. - Text compares under the constraint's collation, from sqlite3_vtab_collation(): BINARY, NOCASE and RTRIM as binCollFunc, nocaseCollatingFunc and rtrimCollFunc do, and any other collation is refused with an error. SQLite names no collation for !=, so vec0 keeps the rows unequal under BINARY and leaves != for SQLite to check again. SQLite orders text under BINARY in the database's encoding, so in a UTF-16 database vec0 leaves <, <=, > and >= on text to SQLite. - A distance compares as the double SQLite receives. The tests compare each filter with a plain table declaring the same types and with SQLite's own scan of the vtab: every operator over NULL, integers, reals, numeric and other text and blobs, texts around the 12 bytes a chunk keeps inline, each collation, IN lists, partition keys, rowid and text id lookups, distances a hair either side of a row's, and UTF-16 databases. A KNN query whose k is the number of matching rows must return all of them, so a filter that keeps any other row fails. The tests fail on main and pass here. The full suite passes (245 passed, 66 skipped, 204 snapshots), also built with AddressSanitizer and UBSan against a SQLite built with AddressSanitizer. make test-unit does not build on main (tests/sqlite-vec-internal.h redeclares the rescore types), and this change adds no error to it.
MayCXC
force-pushed
the
constraint-affinity
branch
from
October 7, 2026 18:16
38c59a5 to
2455320
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vec0 checks a KNN query's constraints itself and sets
aConstraintUsage[].omit, which tells SQLite a constraint "is assumed to be fully handled by the virtual table and might not be checked again" (sqlite3.h). Each constraint vec0 claims therefore has to keep exactly the rows SQLite would keep for the same WHERE clause, and today the same WHERE clause keeps different rows with and without amatch:On main:
n = ?bound to NULL or'abc'keeps n = 0,n = 5.5keeps n = 5,n < 5.5drops n = 5, andn in (5.5, 6)keeps n = 5.f > 'abc'keeps every positive f, though every number is less than any text.b = 2keeps the true rows.t in (null)keeps t = '', andt = x'78'keeps t = 'x'.t < ?bound to NULL crashes (a NULL pointer read) once a chunk holds a text longer than 12 bytes, and<,<=,>and>=fail with "Could not filter metadata fields" once a chunk holds a text of exactly 12 bytes.t = 'abc' COLLATE NOCASEmisses 'ABC', andCOLLATE RTRIMmisses 'abc '.p = '5'misses p = 5,s = 5misses s = '5', and collations are ignored.rowid in (5.5, 'abc', null)matches rowids 5 and 0, and a point lookuprowid = ?matches rowid 0 for NULL,'abc'and a blob, and rowid 5 for 5.5. With a text primary key, an id inid in (...)that names no row empties the whole result.distance > ?bound to NULL keeps every row anddistance < 'abc'none, and distances compared in f32 (the TODO beside that cast) keep the row at distance 2 fordistance <= 1.999999999999, which SQLite drops.This follows SQLite's R-Tree, the extension that checks its own constraints:
rtreeInit()does: metadata columnsboolean,integer,floatortext, partition keys and a text primary key with theirs,rowid integeranddistance real. SQLite's own checks then give a value the affinity vec0 gives it.sqlite3IntFloatCompare()does, every number sorts before any text, and every text before any blob. A rowid lookup takes the value asrtreeFilter()does: numeric affinity, then an integer or a real equal to one.sqlite3_vtab_collation(): BINARY, NOCASE and RTRIM asbinCollFunc,nocaseCollatingFuncandrtrimCollFuncdo, and any other collation is refused with an error. SQLite names no collation for!=(it hands it over without its left operand), so vec0 keeps the rows unequal under BINARY and leaves!=for SQLite to check again. SQLite orders text under BINARY in the database's encoding, so in a UTF-16 database vec0 leaves<,<=,>and>=on text to SQLite.The tests compare each filter with a plain table declaring the same types and with SQLite's own scan of the vtab: every operator over NULL, integers, reals, numeric and other text and blobs, texts around the 12 bytes a chunk keeps inline, each collation,
in (...)lists, partition keys, rowid and text id lookups, distances a hair either side of a row's, and UTF-16 databases. A KNN query whosekis the number of matching rows must return all of them, so a filter that keeps any other row fails. The tests fail on main and pass here. The full suite passes (245 passed, 66 skipped, 204 snapshots), also built with AddressSanitizer and UBSan against a SQLite built with AddressSanitizer.make test-unitdoes not build on main (tests/sqlite-vec-internal.hredeclares the rescore types), and this change adds no error to it.The same change is open on vlasky's fork as vlasky#12, which also covers its LIKE, GLOB, IS and IS NOT filters.