Hi All, I am using pinot libraries to load a segme...
# general
s
Hi All, I am using pinot libraries to load a segment and modify the value of a column in a table to add additional encrypted data for other workflow purpose. As part of this task, i am using pinot libraries and its API's. Once the column values are modified, recreate the segment as csv file format and convert it back to segments again. The updated segments, i would be using PINOT REST APIs to upload the same in PINOT DB. Below are the libraries used. implementation 'org.apache.pinotpinot common1.3.0' implementation 'org.apache.pinotpinot segment spi1.3.0' implementation 'org.apache.pinotpinot segment local1.3.0' implementation 'org.apache.pinotpinot core1.3.0' implementation 'org.apache.pinotpinot spi1.3.0' implementation("org.reflectionsreflections0.10.2") IndexSegment indexSegment = ImmutableSegmentLoader.load(segmentFileToLoad, ReadMode.mmap; / IndexSegment indexSegment = ImmutableSegmentLoader.load(segmentFileToLoad, ReadMode.heap); For the above API has been invoked from Spring 6, SPRING Boot Microservices Simple Test Program. The output of above method is not returning the indexSegment. Further debugged the pinot code, i am getting the error in below lines... pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/PinotDataBuffer.java. /** * Memory maps a file into a buffer. * pNOTE: If the file gets extended, the contents of the extended portion of the file are not defined. */ public static *PinotDataBuffer mapFile*(File file, boolean readOnly, long offset, long size, ByteOrder byteOrder, /** * Allocates a buffer using direct memory and loads a file into the buffer. */ public static PinotDataBuffer loadFile(File file, long offset, long size, ByteOrder byteOrder, @Nullable String description) throws IOException { PinotDataBuffer buffer; Above line is invoked from "*pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java*" Code is stuck in loadFile(...), mapFile(....) methods. private void mapAndSliceFile(SortedMapLong, IndexEntry startOffsets, ListLong offsetAccum, long endOffset) throws IOException { Preconditions.checkNotNull(startOffsets); Preconditions.checkNotNull(offsetAccum); Preconditions.checkArgument(!offsetAccum.isEmpty()); long fromFilePos = offsetAccum.get(0); long size = endOffset - fromFilePos; String context = allocationContext(_indexFile, "single_file_index.rw." + "." + String.valueOf(fromFilePos) + "." + String.valueOf(size)); // Backward-compatible: index file is always big-endian PinotDataBuffer buffer; if (_readMode == ReadMode.heap) { buffer = *PinotDataBuffer.loadFile(_*indexFile, fromFilePos, size, ByteOrder.BIG_ENDIAN, context); } else { buffer = *PinotDataBuffer.mapFile*(_indexFile, true, fromFilePos, size, ByteOrder.BIG_ENDIAN, context); } The same program works fine in Spring framework 3, 4 and only spring framework 6, spring 3 it is not working. Need some inputs on spring framework 6 the same jars with API's are not working? Added below add-opens JVM as well in spring framework 6.... --add-opens java.base/java.lang=ALL-UNNAMED \ --add-opens java.base/java.util=ALL-UNNAMED \ --add-opens java.base/java.nio=ALL-UNNAMED \ --add-opens java.base/java.io=ALL-UNNAMED \ --add-opens java.base/java.security=ALL-UNNAMED \ --add-opens java.base/sun.nio.ch=ALL-UNNAMED \ --add-opens java.base/java.lang.reflect=ALL-UNNAMED \ --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
x
is it java 21?
s
Java 17 with Spring 6
Thanks. I tried this option as well.. --add-opens=java.base/jdk.internal.ref=ALL-UNKNOWN. But still spring6 is not allowing to invoke the API (PinotDataBuffer.mapFile(...))!!! Also the same class, the jar pinot-spi.jar 1.3.0, pinto-local-spi -> 1.3.0 jars were used in Spring 5 without any issues. But in Spring6, it demands me to add org.relections 0.10.2 in order work with pinot-spi.jar pinot-local-spi jars... I have added as part of my spring 6 project jar as a gradle dependency and then it went to a level where PinotDataBuffer.mapFile(...) Abstract class with static method invocation by our internal class as below. Reference: ./pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/memory/PinotDataBuffer.java /** * Memory maps a file into a buffer. * pNOTE: If the file gets extended, the contents of the extended portion of the file are not defined. */ public static PinotDataBuffer mapFile(File file, boolean readOnly, long offset, long size, ByteOrder byteOrder, @Nullable String description) throws IOException { pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java. Above class invokes the PinotDataBuffer.mapFile(...) method // Backward-compatible: index file is always big-endian PinotDataBuffer appendBuffer = *PinotDataBuffer.mapFile(*_indexFile, false, entry._startOffset, entry._size, ByteOrder.BIG_ENDIAN, allocContext); It got failed abruptly and the classloader is not loading it seems in Spring 6....
x
then i think it's the spring6 classloader issue
can you try to Disable layered classloader, Disable devtools classloader, Disable AOT
then run
System.out.println(PinotDataBuffer.class.getClassLoader());
and check if you see
jdk.internal.loader.ClassLoaders$AppClassLoader
s
I see below issue, Still the above changes are not working... It did not print anything .... LOGGER.error("SURESHPERUMAL SFID :: 31B Classloader {} ", PinotDataBuffer.class.getClassLoader()); The moment i hit the above, i dont get the logs itself. build.gradle bootJar { layered = false } tasks.named("processAot") { enabled = false } tasks.named("compileAot") { enabled = false } tasks.named("testAot") { enabled = false } application.properties spring.devtools.restart.enabled=false spring.devtools.restart.exclude=** spring.devtools.livereload.enabled=false spring.classloader.use-original-loader=true JVM change -Dspring.devtools.restart.enabled=false
x
Then you need to check how spring can work with Pinot mmap and unsafe.