Zsolt Takacs
03/26/2021, 9:32 AMMayank
Kevin P
03/29/2021, 11:28 PMJosh Highley
03/30/2021, 9:27 PMIf there are multiple controllers, Pinot expects that all of them are configured with the same back-end storage system so that they have a common view of the segments (e.g. NFS). Pinot can use other storage systems such as HDFS or ADLS
I can't find more info about this. Would any mountable NFS work? S3 for example?Akash
03/31/2021, 10:56 AMDianjin Wang
04/01/2021, 3:16 AMSrini Kadamati
04/01/2021, 1:25 PMTing Chen
04/02/2021, 12:27 AMOguzhan Mangir
04/02/2021, 1:53 PMjobSpecFile
config for reading job spec instead of local path?
${SPARK_HOME}/bin/spark-submit \\
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \\
--master "local[2]" \\
--deploy-mode client \\
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \\
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \\
local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \\
-jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/sparkIngestionJobSpec.yaml
like;
-jobSpecFile <hdfs://bucket/pinot-specs/sparkIngestionJobSpec.yaml>
Kevin Vu
04/02/2021, 4:25 PMPhil Fleischer
04/02/2021, 5:42 PMMike Davis
04/02/2021, 5:55 PMKarin Wolok
Oguzhan Mangir
04/06/2021, 6:45 PMMaheshdatta Mishra
04/08/2021, 11:38 AMJosh Highley
04/08/2021, 9:14 PMRicardo Bernardino
04/09/2021, 8:32 AMZsolt Takacs
04/09/2021, 12:20 PMHIGHEST_KAFKA_OFFSET_CONSUMED
or HIGHEST_STREAM_OFFSET_CONSUMED
metric for monitoring stream ingestion lag, and found that it was removed as part of https://github.com/apache/incubator-pinot/issues/5359
Is there an alternative way to monitor the stream ingestion?Vaibhav Sinha
04/09/2021, 1:52 PMAaron Wishnick
04/09/2021, 3:54 PMChad Preisler
04/09/2021, 10:01 PMPrashant Kumar
04/12/2021, 8:55 PMAkash
04/13/2021, 2:51 PMbin/start-server.sh -zkAddress pinot1.plan:2181,pinot2.plan:2181,pinot3.plan:2181 -configFileName conf/server.conf
and server config are
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.server.storage.factory.hdfs.hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.dataDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.6.0-bin/data/PinotServer/index
pinot.server.instance.segmentTarDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.6.0-bin/data/PinotServer/segmentTar
Akash
04/13/2021, 2:53 PM-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-hdfs
Oguzhan Mangir
04/13/2021, 3:23 PMAaron Wishnick
04/13/2021, 3:54 PMAaron Wishnick
04/13/2021, 3:56 PMTing Chen
04/13/2021, 5:51 PMAaron Wishnick
04/13/2021, 7:13 PMAaron Wishnick
04/13/2021, 7:29 PMfunctionColumnPairs
, will AVG
implicitly be able to use the star tree index or do I need to put AVG
in that list too?