https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • a
    Copy code
    You will also need to provide proper Hadoop dependencies jars from your Hadoop installation to your Pinot startup scripts.
  • k
    @Kevin Johnson has left the channel
  • a
    Looks like there is a huge lack of documentations on HDFS integration in pinot,
    Copy code
    [akashmishra@hotpanel1.mlan ClickGift]$ /local/spark23/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand  --master yarn --deploy-mode client  --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dplugins.include=pinot-s3,pinot-parquet,pinot-hdfs -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml"  --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-shaded.jar"  local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar -jobSpecFile batch-job-spec.yml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/lib/pinot-all-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-0.7.1-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/local/spark23/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2021/05/05 22:22:04.724 WARN [NativeCodeLoader] [main] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021/05/05 22:22:04.770 INFO [SecurityManager] [main] Changing view acls to: akashmishra
2021/05/05 22:22:04.771 INFO [SecurityManager] [main] Changing modify acls to: akashmishra
2021/05/05 22:22:04.771 INFO [SecurityManager] [main] Changing view acls groups to:
2021/05/05 22:22:04.772 INFO [SecurityManager] [main] Changing modify acls groups to:
2021/05/05 22:22:04.772 INFO [SecurityManager] [main] SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(akashmishra); groups with view permissions: Set(); users  with modify permissions: Set(akashmishra); groups with modify permissions: Set()
2021/05/05 22:22:05.171 INFO [IngestionJobLauncher] [main] SegmentGenerationJobSpec:
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
cleanUpOutputDir: false
excludeFileNamePattern: null
executionFrameworkSpec:
  extraConfigs: {stagingDir: '<hdfs://bihadoop/tmp/users/akashmishra/pinot_staging/>'}
  name: spark
  segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner
  segmentMetadataPushJobRunnerClassName: null
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentTarPushJobRunner
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentUriPushJobRunner
includeFileNamePattern: null
inputDirURI: <hdfs://bihadoop/tmp/users/akashmishra/hotpanel_split_event/ClickGift/dt=2020-12-30/>
jobType: SegmentCreationAndTarPush
outputDirURI: <hdfs://bihadoop/tmp/users/akashmishra/pinot_segment/ClickGift/dt=2020-12-30/>
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: '<http://hotpanel1.mlan:9000>'}
pinotFSSpecs:
- className: org.apache.pinot.plugin.filesystem.HadoopPinotFS
  configs: {hadoop.conf.path: /local/hadoop/etc/hadoop/}
  scheme: hdfs
pushJobSpec: {pushAttempts: 2, pushParallelism: 2, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader,
  configClassName: null, configs: null, dataFormat: parquet}
segmentCreationJobParallelism: 0
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: null, tableConfigURI: null, tableName: ClickGift}
tlsSpec: null

2021/05/05 22:22:05.172 INFO [IngestionJobLauncher] [main] Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner
2021/05/05 22:22:05.190 INFO [PinotFSFactory] [main] Initializing PinotFS for scheme hdfs, classname org.apache.pinot.plugin.filesystem.HadoopPinotFS
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.get(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)Lorg/apache/hadoop/tracing/SpanReceiverHost;
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
	at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67)
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:54)
	at org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner.run(SparkSegmentGenerationJobRunner.java:131)
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:142)
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:113)
	at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:132)
	at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main(LaunchDataIngestionJobCommand.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021/05/05 22:22:05.356 INFO [ShutdownHookManager] [Thread-1] Shutdown hook called
2021/05/05 22:22:05.357 INFO [ShutdownHookManager] [Thread-1] Deleting directory /tmp/spark-37bce53b-0d85-4076-a667-1a11cd8b289b
  • c
    I was try to do 
    mvn install
    on pinot repo, Pinot Spark Connector is failing to build [is there any docs on building Spark Connector?]
    Copy code
    [INFO] Pinot Connectors ................................... SUCCESS [  0.229 s]
[INFO] Pinot Spark Connector .............................. FAILURE [  7.478 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:23 min (Wall Clock)
[INFO] Finished at: 2021-05-06T13:36:03+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project pinot-spark-connector: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 240 (Exit value: 240) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException>
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :pinot-spark-connector

 ~/Work/Github/pinot/incubator-pinot  helm-sec-update  mvnd clean install -DskipTests -Dfast
  • x
    Can you check scala version ?
  • j
    @Xiang Fu Can u approve the workflow again for PR : https://github.com/apache/incubator-pinot/pull/6842 Thanks in advance 🙂
  • j
    Hello What is the recommended approach to getting the "last non-null value" ? Use a UDF ?
  • p
    Hello, does anyone know the meaning of the following warn log in the server: 
    2021/05/10 09:11:48.182 WARN [LLRealtimeSegmentDataManager_HitExecutionView__4__5__20210430T1719Z] [HitExecutionView__10__4__20210429T1719Z] Commit failed  with response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":false,"status":"FAILED","offset":-1}
    This is a realtime table, consuming messages from kafka with upsert enabled. I've noticed that the table hasn't been updating offsets for the past 3 days.
  • t
    We’re also running into a strange problem with hybrid tables that might be connected to completely empty realtime table(part): querying the hybrid table gives no records while querying the offline table(part) does. Filtering the time column to an interval for records in the offline table doesn’t help either. Failed to find any relevant log message that can help me identify, what could be the issue.
  • c
    Hi All, When i using pinot0.7.1 I found this error in log
  • c
    Do you know how to dix it, thx
  • r
    Is there any option to remove dead server from pinot UI.
  • a
    Hi Team -- I am trying to test some simple joins with presto . Seeing this issue for one of the tables even when I just do a select * on that table.
    Copy code
    java.lang.NullPointerException: null value in entry: Server_172.18.0.3_7000=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:32)
	at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:42)
	at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:72)
	at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:124)
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:458)
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:437)
	at com.facebook.presto.pinot.PinotSegmentPageSource.queryPinot(PinotSegmentPageSource.java:242)
	at com.facebook.presto.pinot.PinotSegmentPageSource.fetchPinotData(PinotSegmentPageSource.java:214)
	at com.facebook.presto.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:161)
	at com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:276)
	at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:241)
	at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
	at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
	at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
	at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
	at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
	at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
	at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:545)
	at com.facebook.presto.$gen.Presto_0_254_SNAPSHOT_2999330____20210512_100627_1.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
  • a
    I have two tables txn (fact) and txtypes (dimension) .. Just added a handful of rows (2 to 3 rows) to check the query plan..
  • a
    but when i query txtypes , i am getting this NPE..
  • a
    on pinot, the select * is working fine for this table.
  • r
    Someone kindly help here. "PINOT UI LOADING FOR A SECOND AND THEN SHOWING BLANK PAGE" Hi starting pinot in cluster mode and started controller , server and broker. Since last 2 weeks it is working fine but today suddenly I am facing this issue.UI is not loading properly it's coming for a second and showing blank. I tried in all the browsers, cleared cache and cookies tried in different laptop but still the issue is same. Kindly suggest.
  • d
    What does the browser dev console has to say? Any error?
  • r
    No error let me attach the short video
  • d
    There is no output in the dev console?
  • r
    No
  • d
    To be sure we are on the same page I’m talking of this:
  • r
    Actually on UI server tab I changed server one of the server host from 8089 to 8008 and now in dev console it's showing same as not found but after that I cleared everything and updated again stated controller server and broker but not sure why it's still looking for same url
  • d
    Can you show a complete screenshot of the first error? It’s cropped
  • d
    Which revision of pinot are you running?
1...146147148...166Latest