@User Is there a particular reason why we chose

MYSQL_ANSI

lex over

CALCITE_DEFAULT

? Does this mean that if i generate SQL using calcite (relbuilder -> relnode -> sqlnode -> sql string), I need to set lex to

MYSQL_ANSI

in order for Pinot to be able to parse the generated query?

I think default behavior of calcite is to treat quoted and unquoted identifiers separately. So if the column name is unquoted, it will convert it into upper case silently. Using MYSQL_ANSI will preserve the case of identifiers whether or not they are quoted -- this is important since since PQL was case-sensitive until few days ago

I remember one reason is about case sensitive and quoting

Also, relbuilder -> relnode -> sqlnode -> sql string will probably not generate a SQL query. RelBuilder is used for building relational expression from a query

<!here> people might want to take a look at this PR -- https://github.com/apache/incubator-pinot/pull/5006

Please see my comment on the PR. For a change of this type (since it changes persistent data), we should at least have a doc in cwiki, wherever, and give people time to think and comment on it.

https://www.linkedin.com/feed/update/urn:li:activity:6625265474131365888/ regarding pinot open source builds

Anyone has a calculator for figuring cost of setting up an Apache Pinot cluster in AWS and/or Azure?

Currently there is no calculator, we are measuring it based on ec2 instances and ebs size

so I am looking at pinot doc and found this line

Results of aggregations with large amounts of group keys (>1M) are approximated

do we have doc about this approximation behavior? like how much it will be approximated?

@User ^^

At every level from segment -> server -> broker, we apply some limits on number of unique group by keys. After those limits are reached, the remaining keys are dropped. These limits are configurable

Screen Shot 2020-01-29 at 3.11.04 PM.png

with order by support in group by, this problem is slightly alleviated. We perform on-the-fly trimming. WHich means, we will first order, and then trim the bottom results. This is better than the current group by approach of blindly trimming after reaching limit.

however, this is only available on server level as of now. At each segment level, we still stop accepting new group by keys after reaching 100k

sounds good, thanks for the explanation. thanks for the detail explanation. do you know how I can help to put this to our https://pinot.readthedocs.io/en/latest/pql_examples.html so other users can also access this?

where did you read about the aggregation being approximated?

from the https://pinot.readthedocs.io/en/latest/pql_examples.html

you could submit a PR, you'll find this in the docs folder

additionally/alternatively, just edit in gitbooks. i believe we're moving to gitbooks for documentation

i think you'll need permissions from Kishore

I added the label called

incompatible

. Please associate this label to PR if you introduce any backward incompatibility. This will help when we come up with the release documentation.

<!here>, I just sent an email to dev@pinot.apache.org to share a doc. I am not sure if everyone has subscribed to mailing list. In any case, here it is:

Hi All, I have shared a doc highlighting a change to TableConfig. It essentially proposes a new model for the users to specify per column encoding and index info along with cleanup in parts of IndexingConfig. As part of text search PR, we need to add another section to IndexingConfig -- something like TextIndexConfig. During discussions with Subbu and Kishore, we thought it is probably the right time to start doing the cleanup and use the new model directly for text search. It would be great if folks can review this soon as this is the last thing that text search PR is waiting on. However, if we realize that this will take some time to get consensus on, then I plan to go ahead with text search PR following the old (existing) model by introducing a new subsection inside IndexingConfig -- that works perfectly fine and in fact that is what PR currently has.  Later when this proposal is approved, we can do the cleanup for everything. https://docs.google.com/document/d/1PyULfbLOYa5OexZMqbP_MZgg-338flKETJOKWS5en3g/edit

<!here> java.lang.RuntimeException: Failed to get table config from push locations: [pinot_host:80] for table: ops_wb_metrics

anyone know the cause of this error?

I take it your are getting this error when trying to push data?