With respect to this how do we specify the jar location and Apache Pinot #troubleshooting

Join Slack

With respect to this, how do we specify the jar lo...

# troubleshooting

Akash

04/20/2021, 5:23 PM

With respect to this, how do we specify the jar location and what jar we need to supply ?

Mayank

04/20/2021, 5:23 PM

@Xiang Fu ^^

Akash

04/20/2021, 5:25 PM

Copy code

export HADOOP_HOME=/local/hadoop/
export HADOOP_CONF_DIR=/local/hadoop/etc/hadoop/
export JAVA_OPTS="-Xms12G -Xmx28G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-server.log -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/plugins/ -Dlog4j2.configurationFile=file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/conf/pinot-server-log4j2.xml"
/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/bin/start-server.sh  -zkAddress hotpanel1.mlan:2181,hotpanel3.mlan:2181,bi24.mlan:2181 -configFileName /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/conf/server.conf

Server Conf

Copy code

pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.dataDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/data/PinotServer/index
pinot.server.instance.segmentTarDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/data/PinotServer/segmentTar

Error which I am getting

Copy code

2021/04/20 17:19:31.724 ERROR [StartServiceManagerCommand] [Start a Pinot [SERVER]] Failed to start a Pinot [SERVER] at 3.081 since launch
java.lang.RuntimeException: java.lang.RuntimeException: Could not initialize HadoopPinotFS
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:74) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.server.starter.helix.SegmentFetcherAndLoader.<init>(SegmentFetcherAndLoader.java:71) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.server.starter.helix.HelixServerStarter.start(HelixServerStarter.java:336) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:150) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:95) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:260) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.access$000(StartServiceManagerCommand.java:57) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:260) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
Caused by: java.lang.RuntimeException: Could not initialize HadoopPinotFS
	at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:71) ~[pinot-hdfs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:54) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	... 9 more
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2644) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) ~[pinot-parquet-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67) ~[pinot-hdfs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:54) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e]
	... 9 more

Xiang Fu

04/20/2021, 5:31 PM

can you try to put to put hadoop jar into your java classpath ?

Xiang Fu

04/20/2021, 5:31 PM

or put it into pinot plugin directory

Akash

04/20/2021, 5:32 PM

which specific jar ?

Akash

04/20/2021, 5:32 PM

hadoop-common ?

Xiang Fu

04/20/2021, 5:32 PM

hadoop common

Akash

04/20/2021, 5:32 PM

There are lots of jars

Akash

04/20/2021, 5:32 PM

Cool. Thanks.

Mayank

04/20/2021, 5:32 PM

Thanks @Xiang Fu.

Mayank

04/20/2021, 5:33 PM

@Akash Could you please confirm if this solves the issue, we can add that into Faq.

Akash

04/20/2021, 6:05 PM

The server script is quitting now.

Copy code

2021/04/20 18:04:01.026 INFO [HelixInstanceDataManagerConfig] [Start a Pinot [SERVER]] InstanceDataManagerConfig, key: reload.consumingsegment , value: false
2021/04/20 18:04:01.026 INFO [HelixInstanceDataManagerConfig] [Start a Pinot [SERVER]] InstanceDataManagerConfig, key: id , value: Server_10.10.211.27_8098
2021/04/20 18:04:01.026 INFO [HelixInstanceDataManagerConfig] [Start a Pinot [SERVER]] InstanceDataManagerConfig, key: readmode , value: mmap
2021/04/20 18:04:01.027 INFO [HelixInstanceDataManager] [Start a Pinot [SERVER]] HelixInstanceDataManagerConfig: Instance Id: Server_10.10.211.27_8098
	Instance Data Dir: /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/data/PinotServer/index
	Instance Segment Tar Dir: /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/data/PinotServer/segmentTar
	Bootstrap Segment Dir: null
	Read Mode: mmap
	Segment format version: null
2021/04/20 18:04:01.029 INFO [SegmentBuildTimeLeaseExtender] [Start a Pinot [SERVER]] Initialized segment build time lease extender executor
2021/04/20 18:04:01.030 INFO [HelixInstanceDataManager] [Start a Pinot [SERVER]] Initialized Helix instance data manager
2021/04/20 18:04:02.382 INFO [FunctionRegistry] [Start a Pinot [SERVER]] Initialized FunctionRegistry with 119 functions: [fromepochminutesbucket, arrayunionint, codepoint, mod, sha256, year, yearofweek, upper, arraycontainsstring, arraydistinctstring, bytestohex, tojsonmapstr, trim, timezoneminute, sqrt, togeometry, normalize, fromepochdays, arraydistinctint, exp, jsonpathlong, yow, toepochhoursrounded, lower, toutf8, concat, ceil, todatetime, jsonpathstring, substr, dayofyear, contains, jsonpatharray, arrayindexofint, fromepochhoursbucket, arrayindexofstring, minus, arrayunionstring, toepochhours, toepochdaysrounded, millisecond, fromepochhours, arrayreversestring, dow, doy, min, toepochsecondsrounded, strpos, jsonpath, tosphericalgeography, fromepochsecondsbucket, max, reverse, hammingdistance, stpoint, abs, timezonehour, toepochseconds, arrayconcatint, quarter, md5, ln, toepochminutes, arraysortstring, replace, strrpos, jsonpathdouble, stastext, second, arraysortint, split, fromepochdaysbucket, lpad, day, toepochminutesrounded, fromdatetime, fromepochseconds, arrayconcatstring, base64encode, ltrim, arraysliceint, chr, sha, plus, base64decode, month, arraycontainsint, toepochminutesbucket, startswith, week, jsonformat, sha512, arrayslicestring, fromepochminutes, remove, dayofmonth, times, hour, rpad, arrayremovestring, now, divide, bigdecimaltobytes, floor, toepochsecondsbucket, toepochdaysbucket, hextobytes, rtrim, length, toepochhoursbucket, bytestobigdecimal, toepochdays, arrayreverseint, datetrunc, minute, round, dayofweek, arrayremoveint, weekofyear] in 1352ms
2021/04/20 18:04:02.383 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing query executor of class: org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl
2021/04/20 18:04:02.387 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Default timeout for query executor : 15000
2021/04/20 18:04:02.387 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Trying to build SegmentPrunerService
2021/04/20 18:04:02.389 INFO [SegmentPrunerService] [Start a Pinot [SERVER]] Adding segment pruner: DataSchemaSegmentPruner
2021/04/20 18:04:02.389 INFO [SegmentPrunerService] [Start a Pinot [SERVER]] Adding segment pruner: ColumnValueSegmentPruner
2021/04/20 18:04:02.389 INFO [SegmentPrunerService] [Start a Pinot [SERVER]] Adding segment pruner: SelectionQuerySegmentPruner
2021/04/20 18:04:02.389 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Trying to build QueryPlanMaker
2021/04/20 18:04:02.390 INFO [InstancePlanMakerImplV2] [Start a Pinot [SERVER]] Initializing plan maker with maxInitialResultHolderCapacity: 10000, numGroupsLimit: 100000
2021/04/20 18:04:02.390 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Trying to build QueryExecutorTimer
2021/04/20 18:04:02.390 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing query scheduler
2021/04/20 18:04:02.393 INFO [QuerySchedulerFactory] [Start a Pinot [SERVER]] Using FCFS query scheduler
2021/04/20 18:04:02.394 INFO [ResourceManager] [Start a Pinot [SERVER]] Initializing with 48 query runner threads and 96 worker threads
2021/04/20 18:04:02.399 INFO [QueryScheduler] [Start a Pinot [SERVER]] Query log max rate: 10000.0
2021/04/20 18:04:02.399 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing Netty query server on port: 8098
2021/04/20 18:04:02.400 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing transform functions
2021/04/20 18:04:02.410 INFO [ServerInstance] [Start a Pinot [SERVER]] Finish initializing server instance
2021/04/20 18:04:02.413 INFO [PinotFSFactory] [Start a Pinot [SERVER]] Did not find any fs classes in the configuration
2021/04/20 18:04:02.414 INFO [PinotFSFactory] [Start a Pinot [SERVER]] Got scheme hdfs, initializing class org.apache.pinot.plugin.filesystem.HadoopPinotFS
2021/04/20 18:04:02.414 INFO [PinotFSFactory] [Start a Pinot [SERVER]] Initializing PinotFS for scheme hdfs, classname org.apache.pinot.plugin.filesystem.HadoopPinotFS

Akash

04/20/2021, 6:05 PM

Error is gone

Akash

04/20/2021, 6:42 PM

@Xiang Fu

Copy code

/usr/bin/java -Xms12G -Xmx28G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-server.log -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/plugins/ -Dlog4j2.configurationFile=file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/conf/pinot-server-log4j2.xml -cp /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/plugins -classpath /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/lib/pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar -Dapp.name=start-server -Dapp.pid=51665 -Dapp.repo=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/lib -Dapp.home=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin -Dbasedir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin org.apache.pinot.tools.admin.PinotServer -zkAddress hotpanel1.mlan:2181,hotpanel3.mlan:2181,bi24.mlan:2181 -configFileName /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/conf/server.conf

I am directly running this command but the App is existing right away

Akash

04/20/2021, 6:43 PM

Quitting*

Akash

04/20/2021, 6:43 PM

Copy code

2021/04/20 18:42:41.346 INFO [InstancePlanMakerImplV2] [Start a Pinot [SERVER]] Initializing plan maker with maxInitialResultHolderCapacity: 10000, numGroupsLimit: 100000
2021/04/20 18:42:41.346 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Trying to build QueryExecutorTimer
2021/04/20 18:42:41.346 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing query scheduler
2021/04/20 18:42:41.348 INFO [QuerySchedulerFactory] [Start a Pinot [SERVER]] Using FCFS query scheduler
2021/04/20 18:42:41.349 INFO [ResourceManager] [Start a Pinot [SERVER]] Initializing with 48 query runner threads and 96 worker threads
2021/04/20 18:42:41.353 INFO [QueryScheduler] [Start a Pinot [SERVER]] Query log max rate: 10000.0
2021/04/20 18:42:41.354 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing Netty query server on port: 8098
2021/04/20 18:42:41.355 INFO [ServerInstance] [Start a Pinot [SERVER]] Initializing transform functions
2021/04/20 18:42:41.364 INFO [ServerInstance] [Start a Pinot [SERVER]] Finish initializing server instance
2021/04/20 18:42:41.367 INFO [PinotFSFactory] [Start a Pinot [SERVER]] Did not find any fs classes in the configuration
2021/04/20 18:42:41.367 INFO [PinotFSFactory] [Start a Pinot [SERVER]] Got scheme hdfs, initializing class org.apache.pinot.plugin.filesystem.HadoopPinotFS
2021/04/20 18:42:41.367 INFO [PinotFSFactory] [Start a Pinot [SERVER]] Initializing PinotFS for scheme hdfs, classname org.apache.pinot.plugin.filesystem.HadoopPinotFS

Akash

04/20/2021, 6:43 PM

After this

Xiang Fu

04/20/2021, 6:47 PM

you mean the server exits after this?

Akash

04/20/2021, 6:47 PM

Yes

Xiang Fu

04/20/2021, 6:47 PM

hmm

Akash

04/20/2021, 6:47 PM

No error

Akash

04/20/2021, 6:47 PM

It just stops.

Xiang Fu

04/20/2021, 6:47 PM

is there console output or pinot-server-logs for the reason?

Akash

04/20/2021, 6:50 PM

The above is console output.

Akash

04/20/2021, 6:50 PM

I will add some more logs at the place its quiting.

Xiang Fu

04/20/2021, 6:50 PM

sure

Xiang Fu

04/20/2021, 6:50 PM

is there any log files on the working directory?

Akash

04/20/2021, 6:52 PM

Copy code

2021/04/20 18:43:58.393 INFO [MBeanRegistrar] [Start a Pinot [SERVER]] MBean HelixCallback:Change=HEALTH,Key=PinotCluster.Server_10.10.211.27_8098,Type=PARTICIPANT has been registered.
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-protobuf 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-kafka-2.0 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-avro 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-distribution 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-csv 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-s3 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-segment-uploader-default 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-yammer 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-batch-ingestion-standalone 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.402 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-confluent-avro 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-thrift 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-orc 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-batch-ingestion-spark 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-gcs 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-hdfs 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-batch-ingestion-hadoop 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-adls 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-json 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-minion-builtin-tasks 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-parquet 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.403 INFO [Utils] [Start a Pinot [SERVER]] Using pinot-segment-writer-file-based 0.7.0-SNAPSHOT-b9b31e5d2ecdc954934cdc1143d6f729b9f22c0e
2021/04/20 18:43:58.408 INFO [DefaultHelixStarterServerConfig] [Start a Pinot [SERVER]] External config key: pinot.server.segment.fetcher.hdfs.class, value: org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
2021/04/20 18:43:58.408 INFO [DefaultHelixStarterServerConfig] [Start a Pinot [SERVER]] External config key: pinot.server.segment.fetcher.protocols, value: file,http,hdfs
2021/04/20 18:43:58.408 INFO [DefaultHelixStarterServerConfig] [Start a Pinot [SERVER]] External config key: pinot.server.instance.enable.split.commit, value: true
2021/04/20 18:43:58.408 INFO [DefaultHelixStarterServerConfig] [Start a Pinot [SERVER]] E

Xiang Fu

04/20/2021, 6:52 PM

from the starter cmd, there seems to be some redundent:

Copy code

-cp /local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.0-SNAPSHOT-bin/plugins

Akash

04/21/2021, 4:42 PM

This is still happening again and agin.

Akash

04/21/2021, 4:42 PM

Server is quitting.

Akash

04/21/2021, 4:43 PM

Copy code

org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();

Akash

04/21/2021, 4:43 PM

On this specific line.

Xiang Fu

04/21/2021, 6:28 PM

hmmm, do you mean this line failed the server?

Akash

04/21/2021, 6:28 PM

Yes. Its very strange.

Xiang Fu

04/21/2021, 6:28 PM

@Ting Chen can you help check what’s your setup

Akash

04/21/2021, 6:30 PM

This happens only when i add hadoop-common jar in classpath.

Akash

04/21/2021, 6:30 PM

Installation works fine without specifying HDFS as deep storage.

Xiang Fu

04/21/2021, 6:31 PM

yeah, i don’t know what could be an issue, we’ve used this in Uber for multiple years

Akash

04/21/2021, 6:31 PM

How are you passing hadoop-common jar dependency in your installation ?

Xiang Fu

04/21/2021, 6:32 PM

either in classpath or put it into pinot-hdfs plugin

Akash

04/21/2021, 6:33 PM

only hadoop-common right or any other hadoop dependency is also required ?

Xiang Fu

04/21/2021, 6:37 PM

hmm, can you try to add more jars

Xiang Fu

04/21/2021, 6:37 PM

hadoop-auth-2.7.1.jar hadoop-mapreduce-client-core-2.7.1.jar hadoop-common-2.7.1.jar

Akash

04/21/2021, 6:38 PM

Sure. Will do.

Xiang Fu

04/21/2021, 6:39 PM

Another way is to add your hadoop classpath to your java classpath

Xiang Fu

04/21/2021, 6:39 PM

so it will load all the hadoop related jars

Akash

04/21/2021, 7:28 PM

I have added lots of jar on plugin dir but Pinot is failing to load it.

Akash

04/21/2021, 7:28 PM

Copy code

2021/04/21 19:26:47.763 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-mapreduce-client-core-2.7.0.jar]
2021/04/21 19:26:47.764 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-common-2.7.0.jar]
2021/04/21 19:26:47.764 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-auth-2.7.0.jar]
2021/04/21 19:26:47.764 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-0.7.1-shaded.jar]
2021/04/21 19:26:47.764 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-hdfs-2.7.0.jar]
2021/04/21 19:26:47.764 ERROR [PluginManager] [main] Failed to load plugin [pinot-hdfs] from dir [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs]
java.lang.ClassCastException: java.net.URL cannot be cast to java.lang.Comparable
	at java.util.ComparableTimSort.countRunAndMakeAscending(ComparableTimSort.java:320) ~[?:1.8.0_275]
	at java.util.ComparableTimSort.sort(ComparableTimSort.java:188) ~[?:1.8.0_275]
	at java.util.Arrays.sort(Arrays.java:1246) ~[?:1.8.0_275]
	at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:195) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:187) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:157) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:123) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:104) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:182) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.tools.admin.PinotBroker.main(PinotBroker.java:35) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
2021/04/21 19:26:47.769 INFO [PluginManager] [main] Trying to load plugin [pinot-hdfs] from location [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs]
2021/04/21 19:26:47.769 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-mapreduce-client-core-2.7.0.jar]
2021/04/21 19:26:47.769 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-common-2.7.0.jar]
2021/04/21 19:26:47.769 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-auth-2.7.0.jar]
2021/04/21 19:26:47.769 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-0.7.1-shaded.jar]
2021/04/21 19:26:47.770 INFO [PluginManager] [main] Successfully loaded plugin [pinot-hdfs] from jar file [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/hadoop-hdfs-2.7.0.jar]
2021/04/21 19:26:47.770 ERROR [PluginManager] [main] Failed to load plugin [pinot-hdfs] from dir [/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs]
java.lang.ClassCastException: java.net.URL cannot be cast to java.lang.Comparable
	at java.util.ComparableTimSort.countRunAndMakeAscending(ComparableTimSort.java:320) ~[?:1.8.0_275]
	at java.util.ComparableTimSort.sort(ComparableTimSort.java:188) ~[?:1.8.0_275]
	at java.util.Arrays.sort(Arrays.java:1246) ~[?:1.8.0_275]
	at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:195) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:187) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:157) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:123) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:104) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:182) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
	at org.apache.pinot.tools.admin.PinotBroker.main(PinotBroker.java:35) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]

Akash

04/21/2021, 7:30 PM

Copy code

[akashmishra@hotpanel1.mlan ~]$ ls -al hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/pinot-file-system/pinot-hdfs/
total 12980
drwxrwxr-x 2 akashmishra wwwaccess    4096 Apr 21 18:46 .
drwxrwxr-x 6 akashmishra wwwaccess    4096 Apr 21 15:43 ..
-rw-r--r-- 1 akashmishra wwwaccess   70600 Apr 21 18:46 hadoop-auth-2.7.0.jar
-rw-r--r-- 1 akashmishra wwwaccess 3425940 Apr 21 18:36 hadoop-common-2.7.0.jar
-rw-r--r-- 1 akashmishra wwwaccess 8235347 Apr 21 18:43 hadoop-hdfs-2.7.0.jar
-rw-r--r-- 1 akashmishra wwwaccess 1528797 Apr 21 18:46 hadoop-mapreduce-client-core-2.7.0.jar
-rw-r--r-- 1 akashmishra wwwaccess    9016 Apr 21 14:06 pinot-hdfs-0.7.1-shaded.jar

Akash

04/21/2021, 7:30 PM

This is the content of plugin dir

Xiang Fu

04/21/2021, 7:31 PM

can you try to put them into java classpath

Akash

04/21/2021, 7:44 PM

Copy code

/usr/bin/java -Xms12G -Xmx28G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-server.log -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/ -Dlog4j2.configurationFile=file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/conf/pinot-server-log4j2.xml -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins -classpath /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/lib/*:/home/akashmishra/hadooplib/hadoop-auth-2.7.0.jar:/home/akashmishra/hadooplib/hadoop-common-2.7.0.jar:/home/akashmishra/hadooplib/hadoop-hdfs-2.7.0.jar:/home/akashmishra/hadooplib/hadoop-mapreduce-client-core-2.7.0.jar -Dapp.name=start-server -Dapp.pid=3319822 -Dapp.repo=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/lib -Dapp.home=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin -Dbasedir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin org.apache.pinot.tools.admin.PinotServer -zkAddress hotpanel1.mlan:2181,hotpanel3.mlan:2181,bi24.mlan:2181 -configFileName /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/conf/server.conf

Akash

04/21/2021, 7:44 PM

Running this command right now. Its started quitting again at the same place

Xiang Fu

04/21/2021, 7:45 PM

hmm, ok, I will take a look

Xiang Fu

04/21/2021, 7:45 PM

is it jdk 8?

Akash

04/21/2021, 7:46 PM

yes

Akash

04/21/2021, 7:46 PM

Copy code

[akashmishra@hotpanel1.mlan ~]$ java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

Xiang Fu

04/21/2021, 7:46 PM

what’s your controller conf

Xiang Fu

04/21/2021, 7:46 PM

it’s also failing right?

Akash

04/21/2021, 7:50 PM

Copy code

controller.data.dir=<hdfs://bihadoop/tmp/users/akashmishra/pinot/data/>
controller.local.temp.dir=<hdfs://bihadooop/tmp/users/akashmishra/pinot/tmp/>
controller.zk.str=hotpanel1.mlan:2181,hotpanel3.mlan:2181,bi24.mlan:2181
controller.enable.split.commit=true
controller.access.protocols.http.port=9000
controller.helix.cluster.name=PinotController
pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.controller.storage.factory.hdfs.hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.controller.segment.fetcher.protocols=file,http,hdfs
pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

Akash

04/21/2021, 7:51 PM

Server conf

Akash

04/21/2021, 7:51 PM

Copy code

pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.dataDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.6.0-bin/data/PinotServer/index
pinot.server.instance.segmentTarDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.6.0-bin/data/PinotServer/segmentTar

Akash

04/21/2021, 7:53 PM

Copy code

/usr/bin/java -Xms8G -Xmx12G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-controller.log -Dplugins.dir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/plugins/ -Dlog4j2.configurationFile=file:/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/conf/pinot-controller-log4j2.xml -classpath /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/lib/*:/home/akashmishra/hadooplib/hadoop-auth-2.7.0.jar:/home/akashmishra/hadooplib/hadoop-common-2.7.0.jar:/home/akashmishra/hadooplib/hadoop-hdfs-2.7.0.jar:/home/akashmishra/hadooplib/hadoop-mapreduce-client-core-2.7.0.jar -Dapp.name=start-controller -Dapp.pid=3339741 -Dapp.repo=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/lib -Dapp.home=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin -Dbasedir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin org.apache.pinot.tools.admin.PinotController -configFileName /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/conf/controller.conf

Akash

04/21/2021, 7:53 PM

This controller command

Akash

04/21/2021, 8:03 PM

https://recordit.co/azilKTOVwB

Xiang Fu

04/21/2021, 9:07 PM

i tired it on my mac and it works fine

Xiang Fu

04/21/2021, 9:07 PM

what’s your os?

Akash

04/21/2021, 9:15 PM

This is crazy, I have added following code in HadoopPinotFS

Copy code

try {
      <http://LOGGER.info|LOGGER.info>("Getting Class org.apache.hadoop.conf.Configuration");
      Class<?> clazz = Class.forName("org.apache.hadoop.conf.Configuration");
      <http://LOGGER.info|LOGGER.info>("Building object");
      Object conf = clazz.newInstance();
    } catch (Exception e) {
      LOGGER.error("Failed instantiating org.apache.hadoop.conf.Configuration " );
      LOGGER.error("Exception is ", e);
    }
    Configuration hadoopConf = new Configuration();
    <http://LOGGER.info|LOGGER.info>("Configuration loaded");

Akash

04/21/2021, 9:16 PM

and it stops here

Copy code

2021/04/21 21:13:49.444 INFO [HadoopPinotFS] [Start a Pinot [SERVER]] Starting to initialized HadoopPinotFS
2021/04/21 21:13:49.444 INFO [HadoopPinotFS] [Start a Pinot [SERVER]] New Config path is  /local/hadoop/etc/hadoop/
2021/04/21 21:13:49.444 INFO [HadoopPinotFS] [Start a Pinot [SERVER]] Getting Class org.apache.hadoop.conf.Configuration
[akashmishra@hotpanel1.mlan ~]$

Akash

04/21/2021, 9:16 PM

Somehow org.apache.hadoop.conf.Configuration is creating some issues when running code.

Akash

04/21/2021, 9:17 PM

Copy code

NAME="CentOS Linux"
VERSION="8"

Xiang Fu

04/21/2021, 9:17 PM

what’s inside: /local/hadoop/etc/hadoop/

Akash

04/21/2021, 9:18 PM

Copy code

[akashmishra@hotpanel1.mlan ~]$ ls -al /local/hadoop/etc/hadoop/
total 188
drwxr-xr-x 2 root root  4096 Apr 12 22:09 .
drwxr-xr-x 3 root root  4096 Apr 12 22:07 ..
-rw-r--r-- 1 root root  4437 Apr 12 22:09 capacity-scheduler.xml
-rw-r--r-- 1 root root  1335 Jan 27 18:28 configuration.xsl
-rw-r--r-- 1 root root   318 Jan 27 18:28 container-executor.cfg
-rw-r--r-- 1 root root   873 Apr 12 22:09 core-site.xml
-rw-r--r-- 1 root root    29 Apr 12 22:09 dfs.exclude
-rw-r--r-- 1 root root  4482 Apr 12 22:09 fair-scheduler.xml
-rw-r--r-- 1 root root  3670 Jan 27 18:28 hadoop-env.cmd
-rw-r--r-- 1 root root  4197 Apr 12 22:09 hadoop-env.sh
-rw-r--r-- 1 root root  2598 Jan 27 18:28 hadoop-metrics2.properties
-rw-r--r-- 1 root root  2490 Jan 27 18:28 hadoop-metrics.properties
-rw-r--r-- 1 root root  9683 Jan 27 18:28 hadoop-policy.xml
-rw-r--r-- 1 root root  6451 Apr 12 22:09 hdfs-site.xml
-rw-r--r-- 1 root root  1449 Jan 27 18:28 httpfs-env.sh
-rw-r--r-- 1 root root  1657 Jan 27 18:28 httpfs-log4j.properties
-rw-r--r-- 1 root root    21 Jan 27 18:28 httpfs-signature.secret
-rw-r--r-- 1 root root   620 Jan 27 18:28 httpfs-site.xml
-rw-r--r-- 1 root root  3518 Jan 27 18:28 kms-acls.xml
-rw-r--r-- 1 root root  1527 Jan 27 18:28 kms-env.sh
-rw-r--r-- 1 root root  1631 Jan 27 18:28 kms-log4j.properties
-rw-r--r-- 1 root root  5511 Jan 27 18:28 kms-site.xml
-rw-r--r-- 1 root root 12259 Apr 12 22:09 log4j.properties
-rw-r--r-- 1 root root   951 Jan 27 18:28 mapred-env.cmd
-rw-r--r-- 1 root root  1418 Apr 12 22:09 mapred-env.sh
-rw-r--r-- 1 root root  4113 Jan 27 18:28 mapred-queues.xml.template
-rw-r--r-- 1 root root  2162 Apr 12 22:09 mapred-site.xml
-rw-r--r-- 1 root root   758 Jan 27 18:28 mapred-site.xml.template
-rw-r--r-- 1 root root    10 Jan 27 18:28 slaves
-rw-r--r-- 1 root root  2316 Jan 27 18:28 ssl-client.xml.example
-rw-r--r-- 1 root root  2268 Jan 27 18:28 ssl-server.xml.example
-rw-r--r-- 1 root root  2250 Jan 27 18:28 yarn-env.cmd
-rw-r--r-- 1 root root  4957 Apr 12 22:09 yarn-env.sh
-rw-r--r-- 1 root root   198 Apr 12 22:09 yarn.exclude
-rw-r--r-- 1 root root  6923 Apr 12 22:09 yarn-site.xml

Akash

04/21/2021, 9:18 PM

All of our Hadoop configuration.

Xiang Fu

04/21/2021, 9:18 PM

is there any logs in your working directory

Xiang Fu

04/21/2021, 9:18 PM

or dump

Xiang Fu

04/21/2021, 9:18 PM

this sudden stopping seems very weird

Akash

04/21/2021, 9:20 PM

Is there any complex shading going on with respect to “org.apache.hadoop.conf.Configuration” ?

Akash

04/21/2021, 9:21 PM

Copy code

org.apache.hadoop.conf.Configurable

This one is loading fine.

Mayank

04/21/2021, 9:21 PM

Side note @Akash @Xiang Fu if you are using latest master, there is now an api to return all relevant configs of pinot configs: https://github.com/apache/incubator-pinot/pull/6817

Xiang Fu

04/21/2021, 9:21 PM

I think Akash is on 0.7.1

Mayank

04/21/2021, 9:22 PM

He did mention he also debugged using master I think

Xiang Fu

04/21/2021, 9:22 PM

i see

Akash

04/21/2021, 9:22 PM

I can deploy current master. I have tried all, master, 0.6.0 and 0.7.1

Xiang Fu

04/21/2021, 9:22 PM

i think the issue here is pinot doesnt start

Xiang Fu

04/21/2021, 9:22 PM

so no api is accessible

Mayank

04/21/2021, 9:23 PM

I see

Akash

04/21/2021, 9:23 PM

Yes. All i can really drill down is that when java is trying to instantiate “org.apache.hadoop.conf.Configuration” then it just stops

Akash

04/21/2021, 9:24 PM

I have even downloaded all the 2.7.0 jars from maven to just be sure and added to classpath.

Xiang Fu

04/21/2021, 10:05 PM

hmmm, @Oguzhan Mangir have you seen similar things when you are trying out the hdfs?

Akash

04/21/2021, 10:30 PM

Copy code

docker run -ti \
    --net=host        \
    --name pinot-server \
    -v /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/conf/:/opt/pinot/configs/ \
    -v /local/hadoop/etc/hadoop:/opt/pinot/hadoopConf \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-server.log" \
    -d apachepinot/pinot:0.7.1 StartServer \
    -zkAddress hotpanel1.mlan:2181,hotpanel3.mlan:2181,bi24.mlan:2181 \
    -configFileName /opt/pinot/configs/server.conf

Akash

04/21/2021, 10:31 PM

How can i pass hadoop jar via Docker ?

Xiang Fu

04/21/2021, 10:31 PM

use -v to mount

Akash

04/21/2021, 10:34 PM

Can i specify

Copy code

-d apachepinot/pinot:0.7.1 StartServer -classpath /opt/pinot/lib/*:/opt/pinot/hadoopjar/*

and it will pick up ?

Akash

04/21/2021, 10:36 PM

In pinot-admin.sh we are already setting classpath

Copy code

exec "$JAVACMD" $ALL_JAVA_OPTS \
  -classpath "$CLASSPATH" \
  -Dapp.name="pinot-admin" \
  -Dapp.pid="$$" \
  -Dapp.repo="$REPO" \
  -Dapp.home="$BASEDIR" \
  -Dbasedir="$BASEDIR" \
  org.apache.pinot.tools.admin.PinotAdministrator \
  "$@"

Akash

04/21/2021, 10:38 PM

Let me ask very dumb question here, how can i set java classpath while running ./pinot-admin.sh ?

Akash

04/21/2021, 11:08 PM

btw even this is crashing

Copy code

docker run -ti \
    --net=host        \
    --name pinot-server \
    -v /home/akashmishra/hpgraph/apache-pinot-incubating-0.7.1-bin/conf/:/opt/pinot/configs/ \
    -v /local/hadoop/etc/hadoop:/opt/pinot/hadoopConf \
    -v /home/akashmishra/hadooplib/hadoop-common-2.7.0.jar:/opt/pinot/hadooplib/hadoop-common-2.7.0.jar \
    -v /home/akashmishra/hadooplib/hadoop-hdfs-2.7.0.jar:/opt/pinot/hadooplib/hadoop-hdfs-2.7.0.jar \
    -e CLASSPATH_PREFIX="/opt/pinot/hadooplib/hadoop-common-2.7.0.jar:/opt/pinot/hadooplib/hadoop-hdfs-2.7.0.jar" \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-server.log" \
    -d <http://dockerio.badoo.com/bi/pinot:0.1|dockerio.badoo.com/bi/pinot:0.1> StartServer \
    -zkAddress hotpanel1.mlan:2181,hotpanel3.mlan:2181,bi24.mlan:2181 \
    -configFileName /opt/pinot/configs/server.conf

Xiang Fu

04/21/2021, 11:15 PM

I think the way you specify classpath is correct

Xiang Fu

04/21/2021, 11:15 PM

I was using something like:

Copy code

CLASSPATH_PREFIX="/Users/xiangfu/.m2/repository/org/apache/hadoop/hadoop-common/2.7.0/hadoop-common-2.7.0.jar:/Users/xiangfu/.m2/repository/com/google/guava/guava/19.0/guava-19.0.jar:/Users/xiangfu/.m2/repository/org/apache/hadoop/hadoop-auth/2.7.0/hadoop-auth-2.7.0.jar" bin/pinot-admin.sh LaunchDataIngestionJob  -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml

Akash

04/21/2021, 11:16 PM

Do we need guava-19.0.jar & hadoop-auth-2.7.0.jar ?

Akash

04/21/2021, 11:41 PM

It’s looks like Pinot needs almost all of the hadoop-common and all dependent jars to run.

Xiang Fu

04/21/2021, 11:52 PM

when we build pinot-hdfs, we make hadoop-common to be provided runtime

Xiang Fu

04/21/2021, 11:53 PM

but still, the issue is why it quits

Xiang Fu

04/21/2021, 11:53 PM

and there is no exception

Xiang Fu

04/21/2021, 11:53 PM

can you check if log4j use sync logger?

Akash

04/22/2021, 12:01 AM

Found the problem.

Akash

04/22/2021, 12:01 AM

Copy code

CLASSPATH_PREFIX="/local/hadoop/share/hadoop/common/*;/local/hadoop/share/hadoop/common/lib/*;"

Akash

04/22/2021, 12:01 AM

It needs all the commons jar and it’s dependencies.

Akash

04/22/2021, 12:02 AM

Will validate again tomorrow and update the docs.

Mayank

04/22/2021, 12:03 AM

🙏🙏

Open in Slack

Previous Next