It is possible that those queries are waiting on the server side long enough, and hit timeout when they start executing

Hey, I am getting the following error.

Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.f$14 of type org.apache.spark.api.java.function.VoidFunction in instance of org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1

I am using

PINOT_VERSION=0.4.0

With the following overridden versions configs to match the environment

<scala.version>2.11.8</scala.version>
<spark.version>2.3.1.tgt.17</spark.version> (which is specific to target)

Env:

Spark version 2.3.1.tgt.17
Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_73

Run Command:

spark-submit \
  --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \
  --master yarn \
  --deploy-mode client \
  --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \
  --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \
  local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \
  -jobSpecFile /home_dir/z00290g/guestSdrGstDataSgl_sparkIngestionJobSpec.yaml

What is the correct schema for a date column ? i am using the following,

{
  "name": "sls_d",
  "dataType": "STRING",
  "format": "1:DAYS:SIMPLE_DATE_FORMAT:yyyy-MM-dd",
  "granularity": "1:DAYS"
}

but i am getting

Caused by: java.lang.IllegalArgumentException: Invalid format: "null"
	at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[pinot-all.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]

do you know the input path

@Kartik Khare or anyoneelse take a look at this MR when you get a chance? https://github.com/apache/incubator-pinot/pull/5755 (Adding default credentials provider, uses common credential provider chain and a bug fix)

we are running pinot on a kubernetes cluster. the pinot server pods takes a lot of time to come up during rolling restart. each pod is taking around 4-5 minutes. Is this time proportional to number of segments in the server? How can I reduce the startup time of the servers?

Hi, I am trying to use

KafkaConfluentSchemaRegistryAvroMessageDecoder

We have a schema registry set up with SSL authentication. I am getting

SSLHandshakeException

Wondering what is the proper way to pass the SSL certs config for the schema registry client? I digged a bit into the code, it seems like pinot needs to update the schema-registry-client to include this (https://github.com/confluentinc/schema-registry/pull/957) with some code changes. Can be accomplished without it too. Wanted to check before if there is an alternative way to accomplish this?

Sorry I am facing another issue while querying the table, after fixing the above issue with this code-change (https://github.com/apache/incubator-pinot/pull/5758). KafkaConsumer seems to be working fine now based on the logs on server node.

Caught exception while processing instance request
java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
        at org.apache.pinot.core.common.datatable.DataTableBuilder.setColumn(DataTableBuilder.java:157) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.query.selection.SelectionOperatorUtils.getDataTableFromRows(SelectionOperatorUtils.java:261) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.operator.blocks.IntermediateResultsBlock.getSelectionResultDataTable(IntermediateResultsBlock.java:348) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.operator.blocks.IntermediateResultsBlock.getDataTable(IntermediateResultsBlock.java:262) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.operator.blocks.InstanceResponseBlock.<init>(InstanceResponseBlock.java:43) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.operator.InstanceResponseOperator.getNextBlock(InstanceResponseOperator.java:37) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.operator.InstanceResponseOperator.getNextBlock(InstanceResponseOperator.java:26) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]
        at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:48) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-39a78f3df43ac613663975844e598f07b18bf623]

FYI, on pinot-0.4.0 , after deleting a segment which was in ERROR state the realtime table is no longer ingesting. Is there an api to restart ingestion?

Untitled

Ok, everything's working now, thanks for all the help, much appreciated!

We wanted to create some grafana dashboards, do you have a generic one that we can use to monitor pinot (i.e. server disk space for segments, jmx counters for qps, etc.)?

Hey, How can I pass auth, during segment pull ? It is over HTTP, hence adding it as user info in the segment uri prefix. But on redirect it does not retain the user info and fails with 401. Any suggestions ?

Hi, i'm trying to receive data using pinot scatter-gather api(

pinot.core.transport.{AsyncQueryResponse, QueryRouter, ServerInstance

) in pinot 0.5.0-snapshot version. i'm running locally now, and using pinot 0.4.0 version in locally now(because when i try to up pinot with the master branch, it can not load the data. there maybe a problem). Can there any problem about backward compatibility? error message;

ERROR org.apache.pinot.core.transport.DataTableHandler - Caught exception while handling response from server: 192.168.2.154_O
java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
	at org.apache.pinot.core.common.datatable.DataTableImplV2.<init>(DataTableImplV2.java:122) ~[classes/:?]
	at org.apache.pinot.core.common.datatable.DataTableFactory.getDataTable(DataTableFactory.java:35) ~[classes/:?]
	at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:67) ~[classes/:?]
	at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:36) ~[classes/:?]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.42.Final.jar:4.1.42.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]

<rant> Is it just me, or Intellij has become a lot more confused loading Pinot (specially when moving between commits), eats up a lot of my time. </rant>

Controller initializes the PinotFs implementation based on the uri, IIRC. So using local dir will instantiate localFs. I am hoping http:// might pick up HDFS, if not, we should fix.

I'm hitting an issue where the offline data I loaded into does not match the query results from Pinot. I run

select * from metrics limit 20000

and I see

numDocsScanned = 15400
totalDocs = 17118

totalDocs matches my raw data size (what I'd expect). The query results match numDocsScanned (which is wrong). I'm happy to share data and schema in a private message.

Yes multi-valued BYTES are not supported yet, iirc

We are using pinot to store log data and noticed that string columns are truncated at 512 characters. Is there another datatype or setting we should use to increase the length?

I'm about to start writing a daily mapreduce to prepare segments for offline ingestion. I'm using Flink for the streaming ingestion. Any design tips for using Flink? • Should I have Flink write the files to S3 and then run LaunchDataIngestionJob using a workflow tool? • What's the status of the batch plugins? Does this make it easy to encapsulate the client-side parts of LaunchDataIngestionJob? https://docs.pinot.apache.org/plugins/pinot-batch-ingestion I'm also fine with writing this in Spark if it makes it a lot easier. I'd prefer Flink to keep the implementation consistent.

I am getting this error while running

RealtimeProvisioningHelperCommand

Also, noticed that we can't do text_match until the text index is created, looks like the latency could be up to 30 mins

For now the workaround is fine for us

I can try to make a change sometime

Hi, When we are running a query individually we are seeing latencies of the order of 3seconds but when we run for example same query parallely we are seeing latencies to be order of 7-10secs for the third query. Do the queries run serially? I have one broker and 2 servers as part of my setup, also would adding more servers be helpful in parallelizing better for optimizing query latencies?

In the docs it just says metric columns "typically" appear in aggregations but in this case dimension columns appear in the aggregations - and we don't see any issue with that. Are there any caveats to aggregating on dimension columns or grouping by metric columns?

Does Pinot have a way to create an alias to a specific table? I'm thinking about the situation where I want to make a large change to a table and I'll need to recreate it. Can I use an alias and do the swap inside Pinot? Or would I want a layer outside of Pinot to convert this alias table name to a specific Pinot table?

where i can find logs for each query? trying to troubleshoot slow queries i tried this but not luck for any of the brokers:

$ kubectl -n pinot logs pinot-broker-6 broker -f

table is required for fetching required information e.g. schema table configs

some more questions on start-tree index, does reload create start-tree for old segments? and how do I verify that start-tree index is generated for the segments? Basically I am not seeing improvement in query times after reload, so wondering if I am missing something