FYI: this is the change that I am looking at for keeping track of precomputed predicates on the Broker side (

QueryOptimizer.optimize

function for

PinotQuery

). Basically, if we can precompute the value of an

Expression

or

Function

, then

precomputed

will contain that value. This will help in adding type support as discussed and also further predicate pruning and optimization in future. Let me know if there are suggestions or alternate ideas for this?

alal@alal-mn1 amrish-pinot-1 % git diff pinot-common/src/thrift/query.thrift
diff --git a/pinot-common/src/thrift/query.thrift b/pinot-common/src/thrift/query.thrift
index 98329f5ac..5445ce645 100644
--- a/pinot-common/src/thrift/query.thrift
+++ b/pinot-common/src/thrift/query.thrift
@@ -47,6 +47,7 @@ struct Expression {
  2: optional Function functionCall;
  3: optional Literal literal;
  4: optional Identifier identifier;
+ 5: optional Literal precomputed;
 }
 
 struct Identifier {
@@ -67,4 +68,5 @@ union Literal {
 struct Function {
  1: required string operator;
  2: optional list<Expression> operands;
+ 3: optional Literal precomputed;
 }

No, we should not have this

precomputed

field

If the result can be pre-computed, we should either remove the predicate or short-circuit the query to directly return

ok, I was planning to do a two-pass expression tree traversal to do mark and remove, but let me see if it can be done in a single pass.

For short-circuit return, we would need to know whether the predicate is evaluating to false, so we would need to keep that information somewhere right?

So for example if the query is

SELECT * from mytable WHERE intColumn = 5.5

, the predicate here will always evaluate to false and hence the query will become

SELECT * from mytable

.

If we don't want to store precomputed values, then one option may be to rewrite the query to

SELECT * from mytable WHERE FALSE

and then short-circuit the query if predicate is

FALSE

? There could be more complicated cases such as

SELECT * FROM mytable WHERE FALSE AND (intColumn > 5 OR FALSE)

. I think this approach will avoid adding precomputed to

Expression

and

Function

, etc while ensuring that we have the information that we need to short-circuit the query. Sounds ok?

The query

SELECT * from mytable WHERE FALSE

isn't a valid query, but its just a temporary form that we are using for optimization so should be ok.

You can remove the predicate only when it evaluates to

true

or it evaluates to

false

as a child under

AND

Short-circuit means we don't even need to send the query to the servers

I'm thinking adding this info (always

true

or

false

) into the return value of the

FilterOptimizer

to pass it back to the caller

We can take small steps. To unblock the issue of parsing error, we can first only convert the integral value to a format without decimal part (

1.0

->

1

)

Let's go with rewriting predicates to TRUE and FALSE, so that

SELECT * from mytable WHERE intColumn = 5.5

, is rewritten to

SELECT * from mytable WHERE FALSE

and that will allow us to do everything including circuiting without adding precomputed.

I think this is a good solution and give us a clean way forward. Later on if someone comes with a better solution, then the existing code will be generic and clean enough to allow for easy refactoring.

My question here is why do we need to send such query to the server?

We don't

🙂

if the predicate is known to be false, then we short-circuit as you mentioned earlier.

You mean add a special filter expression as

FALSE

?

Yes sir, for example in case of query

SELECT * FROM mytable WHERE FALSE

and if it the query gets rewritten to

SELECT * FROM mytable WHERE TRUE

then we drop the WHERE clause.

Yeah, that works

Currently we don't support this syntax, but I think it is valid sql

Yes, I think in some database (mysql if I am not mistaken) its valid.

cool 🙂 I think we are on the same page @User ^^

To summarize, we want to do the followings: 1. Introduce

TRUE

and

FALSE

as valid predicate 2. Implement the filter optimizer to rewrite values 3. Short circuit the query if the predicate is

FALSE

I like the idea of adding predicate

TRUE

and

FALSE

to pass around the information

Yes, but I would qualify point 1 slightly to read

Introduce TRUE and FALSE as valid predicate in Broker optimizer.

As a separate ticket item, we could add SQL querying support for queries that contain TRUE/FALSE in WHERE clause.

Sounds good

ok cool 🙂

@User has left the channel