Repository navigation
[Java] Implement arrow-memory-ffm #163
Description
Describe the enhancement requested
Add a memory module that uses the java.lang.foreign module (currently in preview, APIs updated in Java 21).
When using Java 9 or later with the current implementations, some JDK internals must be exposed by adding --add-opens=java.base/java.nio=ALL-UNNAMED to the java command[1]. Here's an article that explains the general Java issue here[2].
Can using java.lang.foreign remove this dependency so users do not have to run with custom options ?
[1]https://arrow.apache.org/docs/java/install.html#java-compatibility
[2]https://community.snowflake.com/s/article/JDBC-Driver-Compatibility-Issue-With-JDK-16-and-Later
Component(s)
Java
Metadata
Metadata
Assignees
Labels
No labels
Activity
lidavidm commentedon Sep 15, 2023
Not until Java 21+ (or really Java 24, when the new memory APIs will presumably be stabilized)
lidavidm commentedon Sep 15, 2023
Also, this is a runtime flag, not a build flag
danepitkin commentedon Sep 15, 2023
Ah I see. Updated to reference runtime now. Shall I close this given this won't happen any time soon?
lidavidm commentedon Sep 15, 2023
The interesting things to me would be (1) implementing an arrow-memory-ffm that uses the experimental API so we can figure out how well it works and (2) refactoring BufferAllocator and other core APIs to be in line with the new upstream APIs so that when Java 24 comes, we're ready
lidavidm commentedon Sep 15, 2023
Because if something is missing for us it's kinda too late to communicate any feedback upstream
[-][Java] Remove dependency on JDK internals in Java 9+[/-][+][Java] Implement arrow-memory-ffm[/+]fb64 commentedon Sep 30, 2026
Hi there, the FFM API is final (since JDK 22, JEP 454) and
sun.misc.Unsafeis on its way out, so I prepared anarrow-memory-ffmmodule along the lines of what is suggested above. I'd like to open a PR if there's interest.Here my branch: https://github.com/fb64/arrow-java/tree/memory-ffm
On the Unsafe side, the memory-access methods were deprecated for removal in JDK 23 (JEP 471), and since JDK 24 the JVM prints a warning on first use (JEP 498). The next phase, planned for "JDK 26 or later", makes them throw
UnsupportedOperationExceptionby default, followed by removal. You can already try that behavior with--sun-misc-unsafe-memory-access=deny. SinceMemoryUtilrelies on Unsafe whichever allocation manager is used, this will affect every Arrow Java user, not onlyarrow-memory-unsafe.It also covers the original request: no more
--add-opens=java.base/java.nio=ALL-UNNAMED.What my branch does:
MemoryUtilbehind a smallMemoryUtilAccessorinterface. The current Unsafe code becomesUnsafeMemoryAccessorand stays the default, so nothing changes for existing users.arrow-memory-ffm(JDK 22+) with an FFM-based accessor and anFfmAllocationManager. Setting-Darrow.allocation.manager.type=FFMswitches both, so Unsafe is not used at all.--add-opensto prove it isn't needed.MemorySegment.reinterpret, which is a restricted method, so--enable-native-access=org.apache.arrow.memory.ffm(orALL-UNNAMEDon the classpath) is needed to avoid a warning (JEP 454 says a future release may turn that warning into an error). That flag is the official opt-in for a public, final API, whereas--add-opensgives reflective access to private JDK fields that can change in any release.ℹ️ Full disclosure: I built this with the help of AI (Claude). Happy to walk through any part of it.
I can split it into two PRs if that's easier to review: the accessor refactoring first, then the new module...