can you file a github issue

we are wrapping most of the join semantics and we will get to distinct/window and others soon

Hi team, I am using the v0.12.0 release build and have been trying to query pinot using v2 engine and I keep getting the following error :

[
  {
    "message": "SQLParsingError:\njava.lang.RuntimeException: Error composing query plan for: select * from order_items join orders on order_items.order_id=orders.id limit 10\n\tat org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:137)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:156)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:131)\n\tat org.apache.pinot.broker.requesthandler.BrokerRequestHandler.handleRequest(BrokerRequestHandler.java:47)\n...\nCaused by: java.lang.UnsupportedOperationException: unsupported!\n\tat org.apache.pinot.query.type.TypeFactory.toRelDataType(TypeFactory.java:82)\n\tat org.apache.pinot.query.type.TypeFactory.createRelDataTypeFromSchema(TypeFactory.java:49)\n\tat org.apache.pinot.query.catalog.PinotTable.getRowType(PinotTable.java:49)\n\tat org.apache.calcite.sql.validate.EmptyScope.resolve_(EmptyScope.java:161)",
    "errorCode": 150
  }
]

I have followed the steps given in the doc but keep getting this error. The same issue was there in v0.11.0. I upgraded it but I am still facing the issue, please help me rectifying it.

Hello Team, some feedback about v2 query engine: 1. "NOT IN" subquery not working 2. Window Function is not working when columns are projected with aggregation window function 3. Ranking Functions not yet supported - row_number(), rank(), dense_rank() etc Adding queries and exceptions about each point as replies to this thread. Would you know if the above functionalities not yet supported (or) there are problems with my cluster setup? Using Pinot version 0.12.1

Hi quick question about V2 multistage engine. I’ve got a single machine acting as a cluster (for sandbox purposes). I’ve tried to enable V2 engine but I keep have

Address already in us

on querry.runner port. As broker / controller and server are using the same machine. Is there any way to make it work on same machine ?

Hello Team, couple of clarification questions regarding v2 query engine: 1. is v2 Query engine enabled by defaultin pinot release version v0.13.0? The documentation states that the v2 engine is in beta-testing phase. Is it production ready with v0.13.0 ? 2. What happens if queries that doesn't require intermediate processing is run on v2 query engine ? is there any performance impact? (or) the v2 engine is intelligent enough to not not perform intermediate processing? 3. Should default engine be used for queries that doesn't require v2 engine? (or) is v2 engine a replacement of v1 engine and the v1 engine will be deprecated going forward ? 4. Does intermediate servers hold table segments (or) they are only used for processing v2 engine queries?

Re: (1) Not yet.

When will lastwithtime, firstwithtime be supported on v2. Looking to sum up the last values asof a time across a set of products without using pandas.

Hello Team, 1. Are joins performed by V2 engine reliable in Pinot v0.12? 2. Also, when will it be released for production purposes? Is it scheduled in near future?

June..

Few Questions about v2 engine - 1. Anyone using it in production? 2. How reliable it is to use for the joins as of now? 3. Once it is released (June I suppose, based on the comments above) , will there be any changes against what beta version as of now offers?

We are targeting June for it to be ready for prod and reliable.

We tried beta version, its not there yet. We are waiting for 13.0 release

Do the indexed get utilised while running on v2 ? Explain plan for v1 shows a range index for me but for same query on v2, it does not

explain plan doesn’t get into details of the index for v2, as v2 explain plan is logical. (v1 explain is physical plan)

Hey team, we suddenly started receiving errors while trying to run queries on V2. V1 still runs good. Error in thread. Any pointers please ? Thanks

Hi I am trying to run the following query:

select PERCENTILETDIGEST(feature_value, 5, 100) as p5
from numerical_features

Using the V2 Multi Stage Query Engine, and I get the following error:

ProcessingException(errorCode:450, message:InternalError:
java.lang.RuntimeException: Error composing query plan for: select PERCENTILETDIGEST(feature_value, 5, 100) as p5
from numerical_features
	at org.apache.pinot.query.QueryEnvironment.getTableNamesForQuery(QueryEnvironment.java:242)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.getMultiStageQueryResponse(PinotQueryResource.java:204)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:170)
...
Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 47: No match found for function signature PERCENTILETDIGEST(<NUMERIC>, <NUMERIC>, <NUMERIC>)
	at jdk.internal.reflect.GeneratedConstructorAccessor161.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
...
Caused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature PERCENTILETDIGEST(<NUMERIC>, <NUMERIC>, <NUMERIC>)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490))

I understand that this function does not exist in the V2 engine, but what is its replacement ?

Hi When using the timestamp index, the following query:

select $timestamp$HOUR 
from categorical_features 
limit 10

It works in the older query engine, but when enabling the V2 engine I receive the following error:

ProcessingException(errorCode:450, message:InternalError:
java.lang.RuntimeException: Error composing query plan for: select $timestamp$HOUR 
from categorical_features limit 10
	at org.apache.pinot.query.QueryEnvironment.getTableNamesForQuery(QueryEnvironment.java:241)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.getMultiStageQueryResponse(PinotQueryResource.java:204)
	at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:170)
...
Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 22: Column '$timestamp$HOUR' not found in any table
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
...
Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column '$timestamp$HOUR' not found in any table
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490))

Is the timestamp index not supported in V2 Engine ?

Hi Another one: I have a timestamp indexed column name "timestamp". When performing the following query:

select distinct "timestamp"
from numerical_features

I get the following error:

[
  {
    "errorCode": 200,
    "message": "QueryExecutionError:\njava.lang.RuntimeException: Error executing query: [0]@pinot-broker-0.pinot-broker-headless.pinot.svc.cluster.local:32855 MAIL_RECEIVE(RANDOM_DISTRIBUTED)\n└── [1]@pinot-server-0.pinot-server-headless.pinot.svc.cluster.local:37275 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@pinot-broker-0.pinot-broker-headless.pinot.svc.cluster.local@{32855,32855}|[0]}\n   └── [1]@pinot-server-0.pinot-server-headless.pinot.svc.cluster.local:37275 AGGREGATE_FINAL\n      └── [1]@pinot-server-0.pinot-server-headless.pinot.svc.cluster.local:37275 MAIL_RECEIVE(HASH_DISTRIBUTED)\n         └── [2]@pinot-server-0.pinot-server-headless.pinot.svc.cluster.local:37275 MAIL_SEND(HASH_DISTRIBUTED)->{[1]@pinot-server-0.pinot-server-headless.pinot.svc.cluster.local@{37275,42193}|[0]}\n...\nCaused by: java.lang.RuntimeException: Received error query execution result block: {1000=class java.sql.Timestamp cannot be cast to class java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform'; java.lang.Long is in module java.base of loader 'bootstrap')\njava.lang.ClassCastException: class java.sql.Timestamp cannot be cast to class java.lang.Long (java.sql.Timestamp is in module java.sql of loader 'platform'; java.lang.Long is in module java.base of loader 'bootstrap')\n\tat org.apache.pinot.common.utils.DataSchema$ColumnDataType.convert(DataSchema.java:354)\n\tat org.apache.pinot.query.runtime.operator.utils.TypeUtils.canonicalizeRow(TypeUtils.java:66)\n\tat org.apache.pinot.query.runtime.operator.MultistageGroupByExecutor.getResult(MultistageGroupByExecutor.java:131)"
  }
]

Right, timestamp type and index is not prioritized in v2 engine right now, we will to support that later.

Hi ! We are evaluating using the v2 engine to perform aggregations across tables in pinot that could result in >5M rows as result. Few questions: • Is it planned at some point to be able to directly save the result into a table ? • If not, would it be possible to have a stream of rows from the broker instead of one big result ? • If not, what would be a way to perform this ? I've looked at the code around the mailbox, currently the PipelineBreakerExecutor fetch all results sync but would it be possible to have an async one ? Thanks

is there a repo for the daily builds? tried this but seems it may not be upto date https://repo.startreedata.io/artifactory/external-snapshots/org/apache/pinot/pinot-distribution/0.13.0-SNAPSHOT/

from the date they are daily build

are you using jars or docker images ?

there are also nightly build images : https://hub.docker.com/r/apachepinot/pinot/tags

some questions/feedback around configuring the multistage engine. i’ll post each separately so convo can be threaded

The only mention of configurations in the docs are

Add the following configurations to your cluster config

. This seems to be lacking. It took me hours to try to understand which needs to be set on servers vs brokers vs both. I ended adding them to the server and broker confs, not the global cluster configurations, but I can’t tell if the global cluster configs would even work.

I needed to set

pinot.query.runner.hostname

because our instance id is different from our hostname. I had to dig through logs to see dns resolution errors and find the config in code. Why do we need this config when we already have

pinot.broker.hostname

?

I tested disabling with

pinot.multistage.engine.enabled=false

. But I noticed that running queries with

useMultistageEngine=true

still works, it just silently uses the v1 engine under the hood. This seems like a problem since it’s not what the user requested, and the actual json response from the broker comes out differently. I would have expected the query to immediately fail

Can any one tell me how to enable pinot v2 engine ? I tried to enable it by passing query Use multiStageEngine=true also checked the v2 engine checkedbox but getting this error in pinot UI Using V2 Multi-Stage Query Engine. This is an experimental feature. Please report any bugs to Apache Pinot Slack channel. ProcessingException(errorCode:450, messageInternalError java.lang.UnsupportedOperationException: V2 Multi-Stage query engine not enabled. Please see https://docs.pinot.apache.org/ for instruction to enable V2 engine. at org.apache.pinot.controller.api.resources.PinotQueryResource.executeSqlQuery(PinotQueryResource.java:143) at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:107) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62))