Ken Krugler
01/27/2021, 9:24 PMPinotFS.listFiles()
implementations not returning the protocol with the path. So you get back /user/hadoop/blah
, not hdfs:///user/hadoop/blah
. When those paths get used later, without knowledge of the file system, then you run into problems. Does anyone know why listFiles()
(and maybe other methods in a PinotFS implementation) don’t include the protocol?Xiang Fu
Xiang Fu
<dependency>
<groupId>org.apache.pinot</groupId>
<artifactId>pinot-batch-ingestion-standalone</artifactId>
<version>${project.version}</version>
<scope>runtime</scope>
</dependency>
into pinot-tools/pom.xmlAmrish Lal
01/31/2021, 5:34 AMselect actor from githubEvents limit 1
which produced the result:
{"id":18542751,"login":"LimeVista","display_login":"LimeVista","gravatar_id":"","url":"<https://api.github.com/users/LimeVista>","avatar_url":"<https://avatars.githubusercontent.com/u/18542751>?"}
then I tried to write a query using JSON_MATCH predicate to match the above row:
select actor from githubEvents WHERE JSON_MATCH(actor, 'login = ''LimeVista''')
and this produces an empty result set. I am wondering if I am missing anything or if the query above is incorrect?Jack
02/03/2021, 10:39 PMGrace Walkuski
02/08/2021, 4:43 PMexecute()
function (no params) to the PinotPreparedStatement. There are already several execute(…)
functions with params, and an executeQuery()
function with no params. Is there a reason this hasn’t been done already? Or why was the executeQuery()
function named differently than the rest? Currently Jooq calls execute()
and it hits this since it hasn’t been overridden. I’m happy to make a PR to add this functionality but wanted to check first. Thanks!Ting Chen
02/12/2021, 12:57 AMkhush
02/22/2021, 8:19 AMKen Krugler
03/04/2021, 4:50 PMDistinctCountHLL
only works for single value fields. It seems like a simple change in DistinctCountHLLAggregationFunction.aggregate()
to check if the BlockValSet
is multi-valued, and if so then call BlockValSet.getXXXMV()
and do a sub-iteration on the secondary array it returns. Does that make sense?Neha Pawar
QuickStart -type OFFLINE
argumentsRavikumar Maddi
03/17/2021, 4:05 AMXiang Fu
Ken Krugler
03/18/2021, 6:01 PMkhush
03/22/2021, 5:23 AMkhush
03/23/2021, 12:25 PMkhush
03/26/2021, 7:02 AMNeha Pawar
Amrish Lal
03/30/2021, 7:13 PMXiang Fu
JDK 11
. After controller disconnected from helix, seems that the this Time task is not stopped as expected and it hangs the test forever. It might not be an issue for prod as this behavior doesn’t happen on normal workflow.
17:37:59.009 [Timer-22] ERROR org.apache.helix.controller.GenericHelixController - Time task failed. Rebalance task type: PeriodicalRebalance, cluster: PinotBrokerRestletResourceStatelessTest
org.apache.helix.HelixException: HelixManager (ZkClient) is not connected. Call HelixManager#connect()
at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:363) ~[helix-core-0.9.8.jar:0.9.8]
at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[helix-core-0.9.8.jar:0.9.8]
at org.apache.helix.controller.GenericHelixController$RebalanceTask.run(GenericHelixController.java:247) [helix-core-0.9.8.jar:0.9.8]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
17:37:59.176 [Timer-83] ERROR org.apache.helix.controller.GenericHelixController - Time task failed. Rebalance task type: PeriodicalRebalance, cluster: PinotControllerModeStatelessTest
org.apache.helix.HelixException: HelixManager (ZkClient) is not connected. Call HelixManager#connect()
at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:363) ~[helix-core-0.9.8.jar:0.9.8]
at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[helix-core-0.9.8.jar:0.9.8]
at org.apache.helix.controller.GenericHelixController$RebalanceTask.run(GenericHelixController.java:247) [helix-core-0.9.8.jar:0.9.8]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
17:37:59.752 [Timer-125] ERROR org.apache.helix.controller.GenericHelixController - Time task failed. Rebalance task type: PeriodicalRebalance, cluster: PinotHelixResourceManagerStatelessTest
org.apache.helix.HelixException: HelixManager (ZkClient) is not connected. Call HelixManager#connect()
at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:363) ~[helix-core-0.9.8.jar:0.9.8]
at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[helix-core-0.9.8.jar:0.9.8]
at org.apache.helix.controller.GenericHelixController$RebalanceTask.run(GenericHelixController.java:247) [helix-core-0.9.8.jar:0.9.8]
at java.util.TimerThread.mainLoop(Timer.java:556) [?:?]
at java.util.TimerThread.run(Timer.java:506) [?:?]
Neha Pawar
<http://localhost:9000/help#/Table/ingestFromFile>
This was working a few weeks ago. Now i get this exception during segment creation
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_282]
at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[?:1.8.0_282]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_282]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_282]
at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_282]
at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_282]
at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_282]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_282]
at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_282]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_282]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_282]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_282]
at org.apache.parquet.HadoopReadOptions$Builder.<init>(HadoopReadOptions.java:95) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.parquet.HadoopReadOptions.builder(HadoopReadOptions.java:79) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.parquet.hadoop.ParquetReader$Builder.<init>(ParquetReader.java:198) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.parquet.avro.AvroParquetReader$Builder.<init>(AvroParquetReader.java:107) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.parquet.avro.AvroParquetReader$Builder.<init>(AvroParquetReader.java:99) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.parquet.avro.AvroParquetReader.builder(AvroParquetReader.java:48) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.plugin.inputformat.parquet.ParquetUtils.getParquetAvroReader(ParquetUtils.java:51) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.plugin.inputformat.parquet.ParquetAvroRecordReader.init(ParquetAvroRecordReader.java:52) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader.init(ParquetRecordReader.java:47) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.spi.data.readers.RecordReaderFactory.getRecordReaderByClass(RecordReaderFactory.java:149) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.getRecordReader(SegmentIndexCreationDriverImpl.java:122) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.init(SegmentIndexCreationDriverImpl.java:98) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.controller.util.FileIngestionUtils.buildSegment(FileIngestionUtils.java:129) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.controller.util.FileIngestionHelper.buildSegmentAndPush(FileIngestionHelper.java:101) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestData(PinotIngestionRestletResource.java:197) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestFromFile(PinotIngestionRestletResource.java:127) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-89a22f097c5ff26396e58950c90d764066a56121]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
Amrish Lal
04/06/2021, 5:36 AM[{"message":"BrokerResourceMissingError","errorCode":410}]
...
Executing command: StopProcess -controller -server -broker -zooKeeper
18885
+ true
18886
+ kill -0 3750
18887
.github/workflows/scripts/.pinot_quickstart.sh: line 71: kill: (3750) - No such process
18888
+ break
18889
+ rm -rf /tmp/PinotAdmin/zkData
18890
+ '[' 0 -eq 0 ']'
18891
Batch Quickstart failed: Cannot get correct result for count star query.
18892
+ echo 'Batch Quickstart failed: Cannot get correct result for count star query.'
18893
+ exit 1
18894
Error: Process completed with exit code 1.
Neha Pawar
Seunghyun
04/14/2021, 7:17 PMMayank
Jack
04/15/2021, 8:36 PMMing Liang
04/15/2021, 10:56 PM[INFO] Reactor Summary for Pinot 0.8.0-SNAPSHOT:
[INFO]
[INFO] Pinot .............................................. SUCCESS [ 15.773 s]
[INFO] Pinot Service Provider Interface ................... SUCCESS [ 2.762 s]
[INFO] Pinot Segment Service Provider Interface ........... SUCCESS [ 1.093 s]
[INFO] Pinot Plugins ...................................... SUCCESS [ 4.414 s]
[INFO] Pinot Metrics ...................................... SUCCESS [ 0.145 s]
[INFO] Pinot Yammer Metrics ............................... SUCCESS [ 4.240 s]
[INFO] Pinot Common ....................................... SUCCESS [ 16.085 s]
[INFO] Pinot Input Format ................................. SUCCESS [ 0.993 s]
[INFO] Pinot Avro Base .................................... SUCCESS [ 1.109 s]
[INFO] Pinot Avro ......................................... SUCCESS [ 0.902 s]
[INFO] Pinot Csv .......................................... SUCCESS [ 0.354 s]
[INFO] Pinot JSON ......................................... SUCCESS [ 0.344 s]
[INFO] Pinot local segment implementations ................ SUCCESS [ 6.542 s]
[INFO] Pinot Core ......................................... SUCCESS [ 9.313 s]
[INFO] Pinot Server ....................................... SUCCESS [ 4.548 s]
[INFO] Pinot Segment Uploader ............................. SUCCESS [ 1.528 s]
[INFO] Pinot Segment Uploader Default ..................... SUCCESS [ 15.425 s]
[INFO] Pinot Controller ................................... SUCCESS [ 50.838 s]
[INFO] Pinot Broker ....................................... SUCCESS [ 4.736 s]
[INFO] Pinot Clients ...................................... SUCCESS [ 0.117 s]
[INFO] Pinot Java Client .................................. SUCCESS [ 0.474 s]
[INFO] Pinot JDBC Client .................................. SUCCESS [ 0.555 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [ 1.501 s]
[INFO] Pinot Batch Ingestion Common ....................... SUCCESS [ 0.393 s]
[INFO] Pinot Minion ....................................... SUCCESS [ 1.718 s]
[INFO] Pinot Confluent Avro ............................... FAILURE [ 0.586 s]
[INFO] Pinot ORC .......................................... SKIPPED
[INFO] Pinot Parquet ...................................... SKIPPED
[INFO] Pinot Thrift ....................................... SKIPPED
[INFO] Pinot Protocol Buffers ............................. SKIPPED
[INFO] Pluggable Pinot file system ........................ SKIPPED
[INFO] Pinot Azure Data Lake Storage ...................... SKIPPED
[INFO] Pinot Hadoop Filesystem ............................ SKIPPED
[INFO] Pinot Google Cloud Storage ......................... SKIPPED
[INFO] Pinot Amazon S3 .................................... SKIPPED
[INFO] Pinot Batch Ingestion for Spark .................... SKIPPED
[INFO] Pinot Batch Ingestion for Hadoop ................... SKIPPED
[INFO] Pinot Batch Ingestion Standalone ................... SKIPPED
[INFO] Pinot Batch Ingestion .............................. SKIPPED
[INFO] Pinot Ingestion Common ............................. SKIPPED
[INFO] Pinot Hadoop ....................................... SKIPPED
[INFO] Pinot Spark ........................................ SKIPPED
[INFO] Pinot Stream Ingestion ............................. SKIPPED
[INFO] Pinot Kafka Base ................................... SKIPPED
[INFO] Pinot Kafka 0.9 .................................... SKIPPED
[INFO] Pinot Kafka 2.0 .................................... SKIPPED
[INFO] Pinot Minion Tasks ................................. SKIPPED
[INFO] Pinot Minion Built-In Tasks ........................ SKIPPED
[INFO] Pinot Segment Writer ............................... SKIPPED
[INFO] Pinot Segment Writer File Based .................... SKIPPED
[INFO] Pinot Tools ........................................ SKIPPED
[INFO] Pinot Integration Tests ............................ SKIPPED
[INFO] Pinot Perf ......................................... SKIPPED
[INFO] Pinot Distribution ................................. SKIPPED
[INFO] Pinot Connectors ................................... SKIPPED
[INFO] Pinot Spark Connector .............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:26 min
[INFO] Finished at: 2021-04-15T15:49:12-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project pinot-confluent-avro: Could not resolve dependencies for project org.apache.pinot:pinot-confluent-avro:jar:0.8.0-SNAPSHOT: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.3.1: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.3.1: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:5.3.1 from/to maven-default-http-blocker (<http://0.0.0.0/>): Blocked mirror for repositories: [confluent (<http://packages.confluent.io/maven/>, default, releases+snapshots)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException>
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :pinot-confluent-avro
➜ incubator-pinot git:(master)
Sharayu Gandhi
04/15/2021, 11:31 PM[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.1:npm (npm run-script build) on project pinot-controller: Failed to run task: 'npm run-script build' failed. (error code 2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException>
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :pinot-controller
Can someone guide me?Phúc Huỳnh
04/16/2021, 2:57 AMAmrish Lal
04/26/2021, 9:30 PMError: Tests run: 9, Failures: 1, Errors: 0, Skipped: 8, Time elapsed: 698.523 s <<< FAILURE! - in org.apache.pinot.integration.tests.FlakyConsumerRealtimeClusterIntegrationTest
Error: org.apache.pinot.integration.tests.FlakyConsumerRealtimeClusterIntegrationTest.setUp Time elapsed: 698.359 s <<< FAILURE!
java.lang.AssertionError: Failed to meet condition in 600000ms, error message: Failed to load 115545 documents
Error: Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 739.3 s <<< FAILURE! - in org.apache.pinot.integration.tests.BasicAuthRealtimeIntegrationTest
Error: org.apache.pinot.integration.tests.BasicAuthRealtimeIntegrationTest.setUp Time elapsed: 725.035 s <<< FAILURE!
java.lang.AssertionError: Failed to meet condition in 600000ms, error message: Failed to load 115545 documents
Xiang Fu