Skip to content

codegraph-server SIGSEGVs at startup on Linux/aarch64 (before main, even --version) #15

Closed
Closed
codegraph-server SIGSEGVs at startup on Linux/aarch64 (before main, even --version)#15

Description

@TueTrustWorks
TueTrustWorks
opened on Aug 7, 2026

Summary

On Linux / aarch64, the codegraph-server binary segfaults deterministically at process startup — every invocation, including codegraph-server --version and --help, which do no real work. The crash is before main(), in a shared library's C++ static-initialization path. Built from source at v0.19.0-7-g06bcc88 (cargo build --release -p codegraph-server).

It is not environment-specific

I reproduced the identical crash with the binary built on two different toolchains, to rule out a bleeding-edge-distro issue:

Build+run env gcc glibc Result
Ubuntu 25.10 (arm64) 15.2 2.43 SIGSEGV at startup
Debian bookworm (arm64) 12 2.36 SIGSEGV at startup

A trivial hello-world Rust binary built on the same images runs fine, so the toolchains themselves are healthy.

Symptom / gdb

$ codegraph-server --version
Segmentation fault (core dumped)   # ~0.1s, rc=139

# gdb: crash before main(), corrupt/unwindable stack
Program received signal SIGSEGV
=> 0x…:  strb  w19, [x0]           # store byte through a pointer loaded from a global
# x0 = garbage (e.g. 0xaaaad51d1a0e); instruction sequence:
#   adrp x0, <page>; ldr x0, [x0, #off]; strb w19, [x0]   (w19 = 1)
# faulting PC is inside a shared lib (libc/libstdc++ region), symbols unavailable

Only ld-linux, libstdc++, libgcc_s, libm, libc are mapped at crash time (ONNX Runtime .so is not loaded for --version).

Ruled out

  • ONNX Runtime — crashes identically whether ort is statically linked (default) or dynamic (fastembed → ort-load-dynamic); the .so isn't even loaded at --version.
  • Static-TLS overflow — TLS MemSiz is ~2KB; bumping GLIBC_TUNABLES=glibc.rtld.optional_static_tls doesn't help.
  • OOM — deterministic, same point every time; not memory-pressure-dependent.
  • Stack overflow — ulimit -s unlimited doesn't help.
  • Environment — crashes under env -i too.

Build notes (in case relevant)

RocksDB (librocksdb-sys 0.16 / RocksDB 8.10) needs CXXFLAGS="-include cstdint" to compile under gcc ≥13; unrelated to the crash but required to build at all on modern toolchains.

Ask

Is codegraph-server expected to work on Linux/aarch64 at this version? Happy to provide a full core dump, LD_DEBUG output, or test a patch. The pre-main store-through-a-bad-global-pointer suggests a static-initializer / relocation issue in a bundled C/C++ dependency.

Activity

  1. anvanster
    added 2 commits that reference this issue on Aug 9, 2026

    fix: prevent aarch64 startup crash and cross-file doc chunk id collis…

    71f0f0f

    feat(release): publish a linux-arm64 engine (#20)

    489ccf1

  2. anvanster commented on Aug 10, 2026

    @anvanster
    anvanster
    on Aug 10, 2026
    Member

    Fixed in v0.20.1.

    Your diagnosis was right, including the mechanism: a store through a bad global pointer during pre-main initialisation. The bug was ours, not your toolchains.

    Root cause

    crates/codegraph-server/src/main.rs carried a compatibility shim for glibc 2.31 (SLES 15 SP4), because ONNX Runtime references __libc_single_threaded, added in glibc 2.32:

    #[no_mangle]
pub static __libc_single_threaded: u8 = 0;   // immutable -> lands in .rodata

    It was declared immutable, so it lands in read-only memory. __libc_single_threaded is a variable glibc writes: it sets the flag during startup and clears it when a thread is created. On aarch64 the symbol is also emitted into .dynsym, and a definition in the executable takes precedence over libc's own, so glibc's startup write bound to our read-only byte and took SIGSEGV before main. That is exactly the strb w19, [x0] with w19 = 1 you captured.

    x86_64 escaped it only because the symbol is not dynamically exported there, so glibc kept using its own copy. Hence "works on x86_64, dies on aarch64" with no other difference, which is why your two-toolchain test correctly ruled out the environment.

    The fix keeps the shim (SLES 15 SP4 still needs something to link against) but gives it writable storage, so glibc owns the value as it expects. There was a second copy of the shim under #[cfg(test)] in lib.rs, so cargo test -p codegraph-server --lib failed the same way; both now share one definition.

    You no longer need to build from source

    The release publishes a codegraph-server-linux-arm64 binary, and all three clients (npm/CLI, VS Code, JetBrains) resolve it. Previously arm64 Linux resolved to nothing, which is why building from source was the only route:

    npm i -g @astudioplus/codegraph-mcp

    Linux requirements, identical for x64 and arm64: glibc >= 2.30 and libstdc++ from GCC 11 or newer (GLIBCXX_3.4.29). The second is the binding one and is not implied by the first, because the engine embeds ONNX Runtime built with GCC 11. Verified running on SLES 15 SP4, Ubuntu 22.04+, Debian 12+, RHEL 9+ and Amazon Linux 2023. It does not run on Ubuntu 20.04, Debian 11, RHEL 8 or Amazon Linux 2, whose libstdc++ is older.

    Verified

    Reproduced your crash on aarch64 Ubuntu 24.04 (glibc 2.39) built from source at the affected commit, exit 139 on --version and --help, then confirmed the fix: symbol moves from R (read-only) to B (writable) and all invocations exit 0. The build now asserts that section class, so this cannot regress silently.

    Thanks for the report. The gdb detail, the two-toolchain comparison and the ruled-out list were what made this findable, and your note about needing CXXFLAGS="-include cstdint" for RocksDB under gcc >= 13 is accurate.

    Please reopen if you still see it on 0.20.1.

  3. anvanster
    closed this as completedon Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions