Skip to content

Build: "--fully-static" configure option is broken #41497

Description

@emerzon

Version

Affects all versions from 12.x to 17.3.1
The test below was done with 16.13.2.

Platform

Linux sagan-02 5.13.0-23-generic #23-Ubuntu SMP Fri Nov 26 11:41:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

./configure --fully-static; make -j8

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

No response

What do you see instead?

g++ -o /home/egomes/a/node-v16.13.2/out/Release/obj.target/libtest_crypto_engine.so -shared -pthread -rdynamic -m64  -Wl,-soname=libtest_crypto_engine.so -Wl,--whole-archive /home/egomes/a/node-v16.13.2/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o -Wl,--no-whole-archive -static
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
collect2: error: ld returned 1 exit status
make[1]: *** [test_crypto_engine.target.mk:123: /home/egomes/a/node-v16.13.2/out/Release/obj.target/libtest_crypto_engine.so] Error 1
make[1]: *** Waiting for unfinished jobs....
rm 9fc6b65b733c27999009f69898f8526185d4c971.intermediate
make: *** [Makefile:110: node] Error 2

Additional information

Somehow seems that the "-shared" flag gets out to the above line when it shouldn't.

Activity

  1. mscdex commented on Jan 13, 2022

    @mscdex
    Contributor

    FWIW if you're using glibc, you'll never get a truly fully static build. You will need to build against a different libc like musl to get that. I've cross compiled a fully static node with musl in the past with success, but I have not tried with recent node branches.

  2. emerzon commented on Jan 13, 2022

    @emerzon
    Author

    FWIW if you're using glibc, you'll never get a truly fully static build. You will need to build against a different libc like musl to get that. I've cross compiled a fully static node with musl in the past with success, but I have not tried with recent node branches.

    That's not correct. I have managed to build fully static binaries with glibc using v11.15.0, for instance:

     ./configure --fully-static --with-intl=full-icu --download=all; make -j$(nproc);
    ...
    egomes@sagan-02:~/node-v11.15.0/out/Release$ file node
    node: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=276ec96b60a15310b903e5d7ae065d544e4ec003, for GNU/Linux 3.2.0, stripped
    

    Apparently this feature broke on v12 and was never really fixed since then.

  3. emerzon commented on Jan 13, 2022

    @emerzon
    Author

    Also, there is #30180 with fix #30199 which apparently was never merged for v13

  4. mscdex commented on Jan 13, 2022

    @mscdex
    Contributor

    I have managed to build fully static binaries with glibc using v11.15.0, for instance:

    What does ldd report for that binary though?

  5. emerzon commented on Jan 13, 2022

    @emerzon
    Author

    I have managed to build fully static binaries with glibc using v11.15.0, for instance:

    What does ldd report for that binary though?

    egomes@sagan-02:~/node-v11.15.0/out/Release$ ldd node
         not a dynamic executable
    
  6. emerzon commented on Jan 13, 2022

    @emerzon
    Author

    There's probably a second issue with the --fully-static switch. Running with make -j1 bypasses the relocation error above, but ends up failing on out/Debug/gen-regexp-special-case just as described in #30180

  7. added
    buildIssues and PRs related to Node.js builds or CI infrastructure.
    on Jan 13, 2022
  8. dna2github commented on Jan 14, 2022

    @dna2github

    current workaround: cd to out folder and manually run the line (remove -static) above failed line (LD_LIBRARY_PATH=/node/out/Release/lib.host:/node/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /node/out/Release/obj/gen/src/regexp; python ../../deps/v8/tools/run.py "/node/out/Release/gen-regexp-special-case" "/node/out/Release/obj/gen/src/regexp/special-case.cc"); then back to src root and continue the build with make again ...

  9. emerzon commented on Jan 14, 2022

    @emerzon
    Author

    current workaround: cd to out folder and manually run the line (remove -static) above failed line (LD_LIBRARY_PATH=/node/out/Release/lib.host:/node/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /node/out/Release/obj/gen/src/regexp; python ../../deps/v8/tools/run.py "/node/out/Release/gen-regexp-special-case" "/node/out/Release/obj/gen/src/regexp/special-case.cc"); then back to src root and continue the build with make again ...

    Thanks for the suggestion!
    I am not quite sure which drawbacks this could have, but it did get me further in the build.

    Now I am facing another (3rd!?) issue, probably related to the static-pie feature present on glibc versions => 2.27.

    make -C out BUILDTYPE=Release V=0
      touch 0d400e6026d015f3b8ab3c23c0a6d3ab4b19fd98.intermediate
      LD_LIBRARY_PATH=/home/egomes/node-v17.3.1/out/Release/lib.host:/home/egomes/node-v17.3.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /home/egomes/node-v17.3.1/out/Release/obj/gen/src/node/inspector/protocol; python tools/inspector_protocol/code_generator.py --jinja_dir tools/inspector_protocol --output_base "/home/egomes/node-v17.3.1/out/Release/obj/gen/src/" --config src/inspector/node_protocol_config.json
      touch 650511246299582ab8a88ef89d6e7eb0a60f7e14.intermediate
      LD_LIBRARY_PATH=/home/egomes/node-v17.3.1/out/Release/lib.host:/home/egomes/node-v17.3.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/objects /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/wasm /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/debug /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/third_party/v8/builtins /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/test/torque /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/builtins /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated/src/ic /home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated; "/home/egomes/node-v17.3.1/out/Release/torque" -o "/home/egomes/node-v17.3.1/out/Release/obj/gen/torque-generated" -v8-root ../../deps/v8 src/builtins/aggregate-error.tq src/builtins/array-at.tq src/builtins/array-concat.tq src/builtins/array-copywithin.tq src/builtins/array-every.tq src/builtins/array-filter.tq src/builtins/array-find.tq src/builtins/array-findindex.tq src/builtins/array-findlast.tq src/builtins/array-findlastindex.tq src/builtins/array-foreach.tq src/builtins/array-from.tq src/builtins/array-isarray.tq src/builtins/array-join.tq src/builtins/array-lastindexof.tq src/builtins/array-map.tq src/builtins/array-of.tq src/builtins/array-reduce-right.tq src/builtins/array-reduce.tq src/builtins/array-reverse.tq src/builtins/array-shift.tq src/builtins/array-slice.tq src/builtins/array-some.tq src/builtins/array-splice.tq src/builtins/array-unshift.tq src/builtins/array.tq src/builtins/arraybuffer.tq src/builtins/base.tq src/builtins/boolean.tq src/builtins/builtins-bigint.tq src/builtins/builtins-string.tq src/builtins/cast.tq src/builtins/collections.tq src/builtins/constructor.tq src/builtins/conversion.tq src/builtins/convert.tq src/builtins/console.tq src/builtins/data-view.tq src/builtins/finalization-registry.tq src/builtins/frames.tq src/builtins/frame-arguments.tq src/builtins/function.tq src/builtins/growable-fixed-array.tq src/builtins/ic-callable.tq src/builtins/ic-dynamic-check-maps.tq src/builtins/ic.tq src/builtins/internal-coverage.tq src/builtins/internal.tq src/builtins/iterator.tq src/builtins/math.tq src/builtins/number.tq src/builtins/object-fromentries.tq src/builtins/object.tq src/builtins/promise-abstract-operations.tq src/builtins/promise-all.tq src/builtins/promise-all-element-closure.tq src/builtins/promise-any.tq src/builtins/promise-constructor.tq src/builtins/promise-finally.tq src/builtins/promise-misc.tq src/builtins/promise-race.tq src/builtins/promise-reaction-job.tq src/builtins/promise-resolve.tq src/builtins/promise-then.tq src/builtins/promise-jobs.tq src/builtins/proxy-constructor.tq src/builtins/proxy-delete-property.tq src/builtins/proxy-get-property.tq src/builtins/proxy-get-prototype-of.tq src/builtins/proxy-has-property.tq src/builtins/proxy-is-extensible.tq src/builtins/proxy-prevent-extensions.tq src/builtins/proxy-revocable.tq src/builtins/proxy-revoke.tq src/builtins/proxy-set-property.tq src/builtins/proxy-set-prototype-of.tq src/builtins/proxy.tq src/builtins/reflect.tq src/builtins/regexp-exec.tq src/builtins/regexp-match-all.tq src/builtins/regexp-match.tq src/builtins/regexp-replace.tq src/builtins/regexp-search.tq src/builtins/regexp-source.tq src/builtins/regexp-split.tq src/builtins/regexp-test.tq src/builtins/regexp.tq src/builtins/string-at.tq src/builtins/string-endswith.tq src/builtins/string-html.tq src/builtins/string-includes.tq src/builtins/string-indexof.tq src/builtins/string-iterator.tq src/builtins/string-match-search.tq src/builtins/string-pad.tq src/builtins/string-repeat.tq src/builtins/string-replaceall.tq src/builtins/string-slice.tq src/builtins/string-startswith.tq src/builtins/string-substr.tq src/builtins/string-substring.tq src/builtins/string-trim.tq src/builtins/symbol.tq src/builtins/torque-internal.tq src/builtins/typed-array-at.tq src/builtins/typed-array-createtypedarray.tq src/builtins/typed-array-every.tq src/builtins/typed-array-entries.tq src/builtins/typed-array-filter.tq src/builtins/typed-array-find.tq src/builtins/typed-array-findindex.tq src/builtins/typed-array-findlast.tq src/builtins/typed-array-findlastindex.tq src/builtins/typed-array-foreach.tq src/builtins/typed-array-from.tq src/builtins/typed-array-keys.tq src/builtins/typed-array-of.tq src/builtins/typed-array-reduce.tq src/builtins/typed-array-reduceright.tq src/builtins/typed-array-set.tq src/builtins/typed-array-slice.tq src/builtins/typed-array-some.tq src/builtins/typed-array-sort.tq src/builtins/typed-array-subarray.tq src/builtins/typed-array-values.tq src/builtins/typed-array.tq src/builtins/weak-ref.tq src/ic/handler-configuration.tq src/objects/allocation-site.tq src/objects/api-callbacks.tq src/objects/arguments.tq src/objects/bigint.tq src/objects/cell.tq src/objects/code.tq src/objects/contexts.tq src/objects/data-handler.tq src/objects/debug-objects.tq src/objects/descriptor-array.tq src/objects/embedder-data-array.tq src/objects/feedback-cell.tq src/objects/feedback-vector.tq src/objects/fixed-array.tq src/objects/foreign.tq src/objects/free-space.tq src/objects/heap-number.tq src/objects/heap-object.tq src/objects/js-array-buffer.tq src/objects/js-array.tq src/objects/js-collection-iterator.tq src/objects/js-collection.tq src/objects/js-function.tq src/objects/js-generator.tq src/objects/js-objects.tq src/objects/js-promise.tq src/objects/js-proxy.tq src/objects/js-regexp-string-iterator.tq src/objects/js-regexp.tq src/objects/js-weak-refs.tq src/objects/literal-objects.tq src/objects/map.tq src/objects/megadom-handler.tq src/objects/microtask.tq src/objects/module.tq src/objects/name.tq src/objects/oddball.tq src/objects/ordered-hash-table.tq src/objects/primitive-heap-object.tq src/objects/promise.tq src/objects/property-array.tq src/objects/property-cell.tq src/objects/property-descriptor-object.tq src/objects/prototype-info.tq src/objects/regexp-match-info.tq src/objects/scope-info.tq src/objects/script.tq src/objects/shared-function-info.tq src/objects/source-text-module.tq src/objects/stack-frame-info.tq src/objects/string.tq src/objects/struct.tq src/objects/swiss-hash-table-helpers.tq src/objects/swiss-name-dictionary.tq src/objects/synthetic-module.tq src/objects/template-objects.tq src/objects/templates.tq src/objects/torque-defined-classes.tq test/torque/test-torque.tq third_party/v8/builtins/array-sort.tq src/objects/intl-objects.tq src/objects/js-break-iterator.tq src/objects/js-collator.tq src/objects/js-date-time-format.tq src/objects/js-display-names.tq src/objects/js-list-format.tq src/objects/js-locale.tq src/objects/js-number-format.tq src/objects/js-plural-rules.tq src/objects/js-relative-time-format.tq src/objects/js-segment-iterator.tq src/objects/js-segmenter.tq src/objects/js-segments.tq src/builtins/wasm.tq src/debug/debug-wasm-objects.tq src/wasm/wasm-objects.tq
      touch 15ebae13038868054c0f7f2ad81251b654d80ee3.intermediate
      LD_LIBRARY_PATH=/home/egomes/node-v17.3.1/out/Release/lib.host:/home/egomes/node-v17.3.1/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/egomes/node-v17.3.1/out/Release/obj/gen/inspector-generated-output-root/include/inspector /home/egomes/node-v17.3.1/out/Release/obj/gen/inspector-generated-output-root/src/inspector/protocol; python ../../deps/v8/third_party/inspector_protocol/code_generator.py --jinja_dir ../../deps/v8/third_party --output_base "/home/egomes/node-v17.3.1/out/Release/obj/gen/inspector-generated-output-root/src/inspector" --config ../../deps/v8/src/inspector/inspector_protocol_config.json --inspector_protocol_dir ../../deps/v8/third_party/inspector_protocol
      clang++ -o /home/egomes/node-v17.3.1/out/Release/obj.target/libtest_crypto_engine.so -shared -pthread -rdynamic -m64  -Wl,-soname=libtest_crypto_engine.so -Wl,--whole-archive /home/egomes/node-v17.3.1/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o -Wl,--no-whole-archive -static -latomic
    /usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 00000000004010e0
    /usr/bin/ld: /home/egomes/node-v17.3.1/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o: in function `bind':
    test_crypto_engine.c:(.text+0x11): undefined reference to `ENGINE_set_id'
    /usr/bin/ld: test_crypto_engine.c:(.text+0x24): undefined reference to `ENGINE_set_name'
    /usr/bin/ld: /home/egomes/node-v17.3.1/out/Release/obj.target/test_crypto_engine/test/fixtures/test_crypto_engine.o: in function `bind_engine':
    test_crypto_engine.c:(.text+0x8e): undefined reference to `ENGINE_get_static_state'
    /usr/bin/ld: test_crypto_engine.c:(.text+0xa4): undefined reference to `CRYPTO_set_mem_functions'
    /usr/bin/ld: test_crypto_engine.c:(.text+0xb3): undefined reference to `ENGINE_set_id'
    /usr/bin/ld: test_crypto_engine.c:(.text+0xc6): undefined reference to `ENGINE_set_name'
    /usr/bin/ld: /lib/x86_64-linux-gnu/libc.a(dl-reloc-static-pie.o): in function `_dl_relocate_static_pie':
    (.text+0x1a): undefined reference to `_DYNAMIC'
    /usr/bin/ld: (.text+0x39): undefined reference to `_DYNAMIC'
    /usr/bin/ld: /home/egomes/node-v17.3.1/out/Release/obj.target/libtest_crypto_engine.so: hidden symbol `_DYNAMIC' isn't defined
    /usr/bin/ld: final link failed: bad value
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[1]: *** [test_crypto_engine.target.mk:126: /home/egomes/node-v17.3.1/out/Release/obj.target/libtest_crypto_engine.so] Error 1
    make[1]: *** Waiting for unfinished jobs....
    rm 0d400e6026d015f3b8ab3c23c0a6d3ab4b19fd98.intermediate 15ebae13038868054c0f7f2ad81251b654d80ee3.intermediate 650511246299582ab8a88ef89d6e7eb0a60f7e14.intermediate
    make: *** [Makefile:110: node] Error 2
    
  10. dna2github commented on Jan 14, 2022

    @dna2github

    yes, you also need to fix out/test_crypto_engine.target.mk to remove -static; then you can get a static node

  11. dna2github commented on Jan 14, 2022

    @dna2github

    and the manual run has no drawbacks since it just builds gen-regexp-special-case binary file and then the host will run it to generate a .cc source file; then its lifecycle is over ...

  12. emerzon commented on Jan 14, 2022

    @emerzon
    Author

    and the manual run has no drawbacks since it just builds gen-regexp-special-case binary file and then the host will run it to generate a .cc source file; then its lifecycle is over ...

    Thanks!!!

    I have then written a scripted workaround for this build, which works for v16 and v17:

    ./configure --fully-static --enable-static --download=all --with-intl=full-icu;
    for i in out/tools/v8_gypfiles/gen-regexp-special-case.target.mk out/test_crypto_engine.target.mk; do 
    sed -i 's/\-static//g' $i || echo "nevermind"; done
    make -j$(nproc)
    

    Which did result in a fully-static binary:

    egomes@sagan-02:~/node-v17.3.1$ ldd out/Release/node
            not a dynamic executable
    egomes@sagan-02:~/node-v17.3.1$ file out/Release/node
    out/Release/node: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=5242bc022d6a7ce2f2e9f363dfb42fc1c5827884, for GNU/Linux 3.2.0, with debug_info, not stripped
    

    Since as you mentioned, these programs are only built to generate another .cpp file, therefore are not part of the actual build, there shouldn't be any drawbacks, however, I am not sure what could be a definitive fix to this.

    Perhaps the build system needs to be patched in order not to pass "-static" (or any other 'runtime' flags) to the helper programs.

    The other approach would be to investigate the segfault on such binaries when they are built with -static.
    This issue seems to be related to symbol ordering issues, as explained in #30180 and also at https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why.
    Apparently, the workaround/solution provided by #30199 works only for older versions of glibc, and not the current one (2.34).

  13. alvarolb commented on Feb 2, 2022

    @alvarolb

    Thank you very much for the script @emerzon !

  14. bnoordhuis commented on Sep 27, 2023

    @bnoordhuis
    Member

    Closing, bug report against EOL version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    buildIssues and PRs related to Node.js builds or CI infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions