codegraph_index_markdown` overwrites previously indexed documents in v0.19.1 #16

Activity

1. anvanster

   added a commit that references this issue on Aug 9, 2026

   fix: prevent aarch64 startup crash and cross-file doc chunk id collis…

   ...

   71f0f0f
2. anvanster commented on Aug 10, 2026

   

   anvanster

   on Aug 10, 2026

   Member

   More actions

   Fixed in v0.20.1.

   Your hypothesis was exactly right, including the cause you suggested: chunk ids restarted at doc-0001 for every source and collided.

   Root cause

   parse_markdown initialised its counter per call, so every document minted doc-0001, doc-0002, and so on. That id is simultaneously:

   • the RocksDB key (doc:{id} and docvec:{id}),
   • the in-memory chunk cache key,
   • the HNSW point id,

   all of which are a single global namespace. So indexing b.md wrote straight over a.md's chunks, and a.md disappeared from codegraph_list_doc_sources and codegraph_search_docs while indexing still reported status: success.

   One detail worth adding to your report, because it explains the "can disappear" you observed rather than consistent loss: the damage is partial and size-dependent. A 3-chunk file overwrites only the first 3 chunks of a 10-chunk file, and the 7 survivors then make the next remove_source look like it did its job. Re-indexing the larger file afterwards wipes the smaller one entirely.

   Fix

   Chunk ids are now namespaced by the source path:

   doc-{fnv1a(source_file):016x}-{counter:04}
   

   FNV-1a rather than the standard library hasher because the value is written into a persisted key, and DefaultHasher's output is explicitly not stable across Rust releases, which would orphan every chunk on disk after a toolchain upgrade.

   Verified end to end with your exact reproduction against a real engine. Before: two files left list_doc_sources reporting 1 source. After: 2 sources, re-indexing the first no longer wipes the second, and both unique-alpha-marker and unique-beta-marker are findable. There is now a regression test for the store-level behaviour as well as the id generation.

   Existing indexes need re-indexing

   Upgrading does not repair data already lost, because the overwrite happened on disk. Old and new ids have different shapes, so they coexist safely and no migration is needed, but any source that was clobbered stays missing until it is indexed again.

   After upgrading, re-run codegraph_index_markdown once per source:

   codegraph_index_markdown(path: "/path/to/a.md")
   codegraph_index_markdown(path: "/path/to/b.md")
   

   Then codegraph_list_doc_sources should report every source, and both should be searchable.

   Thanks for the report. The minimal reproduction and the chunk-id observation pointed straight at the defect.

   Please reopen if anything is still missing after a re-index on 0.20.1.

   👍React with 👍1Reacted by finix7776
3. anvanster

   closed this as completedon Aug 16, 2026