Apache Pinot

Haha
I'm still trying to figure out how it would fit, but if I find a way to use star tree in my usecase, maybe we could add a similar example in the docs ?
(as mine is quite different from the current one)

Thanks! - I made a PR - <https://github.com/pinot-contrib/pinot-docs/pull/39>

<https://github.com/pinot-contrib/pinot-docs/pull/42> a faq about when data becomes queryable while being ingested into Realtime table, pls review

thanks for the review! just updated. pls check it out

kindly reminder to help review the PR again :thankyou:

My doc PR above got merged, but I am not seeing the changes on the web. I am wondering if there is anything else that I need to do?

you may need to add an entry in the SUMMARY.md file like
```* [Query](users/user-guide-query/README.md)
...
  * [Lookup UDF Join](users/user-guide-query/lookup-udf-join.md)
  * [JSON Queries]... &lt;---```


quick question, is this new feature related to JsonExtractScalar functions <https://docs.pinot.apache.org/users/user-guide-query/supported-transformations#json-functions|mentioned in here>?

its built on top of jsonextractscalar, jsonmatch, and jsonindex.

uses all three depending on whether there is json index on the column or not.

ack. I assume the query parser checks if the column is JSON type, if so, it tries to wrap the json path expressions with relevant json functions?

So I updated the SUMMARY.md file, but the information is still not showing up in <https://docs.pinot.apache.org/users/user-guide-query>

image.png

ahh, i see it now. :slightly_smiling_face:

np, thanks for adding the feature and docs :thumbsup:

Or did you see a release announcement that I missed?

I'm talking about the release on Github
<https://github.com/apache/pinot/releases/tag/release-0.8.0>

Yeah, that is still under voting and not officially announced. Once release notes are ready it will be made public.

Hi folks, a QQ, I tried to make changes to Pinot-docs, but somehow the modified MD file is shown as Binary. Any idea why this happened? <https://github.com/pinot-contrib/pinot-docs/pull/54>

Check this to see if it applies: <https://stackoverflow.com/questions/777949/can-i-make-git-recognize-a-utf-16-file-as-text>

Thanks <@UDQU92KBK> . I think it is due to UTF16 chars

Hi <@UDT7GFEG6>, I found that other MD files are shown as text format but only table.md is shown as binary.  For example this <https://github.com/pinot-contrib/pinot-docs/commit/32d3c5fe2bafd1e42770dfcae4bef4f14e179928#diff-151da263f43456f02463b1452049a696feb646fd00052884a26aa1195c71b3d2|PR>,

Seems I also cannot use iconv to convert the encoding to utf-8,
```iconv: configuration-reference/table.md:1:24285: incomplete character or shift sequence```
Probably because of some of the previous changes.

<@U01FBQ90RT6> Can you please rebase the change and see if it can correctly show

It works now. Thanks for fixing it. I also updated my PR.

Hi all (and specifically <@UDT7GFEG6>) - I was talking about `ID_SET` support in Pinot with co-workers, and tried to point them at the relevant documentation…which I couldn’t find (also `IN_ID_SET`, not sure what else might be relevant). Does this exist somewhere that Google can’t see?

I did find <https://docs.google.com/document/d/1s6DZ9eTPqH7vaKQlPjKiWb_OBC3hkkEGICIzcd5gozc/edit#>, but that’s not end-user documentation.

on my list of things to do! <https://github.com/apache/pinot/issues/7789>

<@U01ECR21TEE> here we go:

<https://docs.pinot.apache.org/users/user-guide-query/querying-pinot#filtering-with-idset>
<https://docs.pinot.apache.org/users/user-guide-query/querying-pinot#sub-query-filtering-with-idset>

I wonder whether we should make a separate page to explain the options for filtering, but for now I've put it on the same page as the other querying examples

Hi <@U01DS9A0DS4> - that’s really good, thanks! Two other items that you might want to cover…
1. I thought that for id set queries other than IN_PARTITIONED_SUBQUERY, you could use an id set from another table, which would be a big win for some use cases. 
2. Also I thought <@UDT7GFEG6> said something about integers being better (faster?) than strings for id sets.

1. cool, good idea, lemme do that
2. when I tried to create an example with strings the serialized IdSet value was insanely large compared to the integer equivalent even with very few values!

For #2, I think that’s because a Bloom filter is used for non-integers, and the size of that is set by these two parameters:
```expectedInsertions (5,000,000 by default): Number of expected insertions for the BloomFilter, must be positive
fpp (0.03 by default): Desired false positive probability for the BloomFilter, must be positive and less than 1.0```
So if you set `expectedInsertions` to say 10, I imagine you’d get a much smaller serialized size.

<@U01ECR21TEE> is correct. For data types other than INT and LONG, we directly start with a bloom filter with 5M expected insertions and 0.03 fpp by default

<@U01DS9A0DS4> Thanks for adding the docs. I think we should also document this behavior and optional arguments so that people can tune the performance accordingly

<@U01ECR21TEE> <@UDT7GFEG6> attempt #2 <https://docs.pinot.apache.org/users/user-guide-query/filtering-with-idset>

Hi <@U01DS9A0DS4> better, thanks. A few comments…
1. Default values for the tuning parameters?
2. Explain that `sizeThresholdInBytes` is used for int &amp; long, and controls when it switches to a Bloom Filter.
3. Note that when a Bloom Filter is used, the filter results are approximate - you can get false positive results (for membership in the set), leading to potentially unexpected results.
4. Cover when the id set can be built from a different table than the one being used for filtering
5. You say “When creating an IdSet for values in non INT/LONG columns, we can configure the expectedInsertions and fpp parameters”, but the example only sets the expectedInsertions. And it’s a bit confusing, in that these parameters would only come into play if the RoaringXXX data structure exceeds the `sizeThresholdInBytes` limit, which has a default of 8MB.
6. You say “The generated IdSet for the first query will be smaller as it will only contain the ids for the partitions served by the server.“. I think it would be clearer if you said “…for the subQuery will be smaller”.

<@UDT7GFEG6> thanks for adding this functionality! In thinking about using this support for one of our use cases, I wish I could configure it to fail if the Roaringxxx data set exceeds the `sizeThresholdInBytes` limit. That way I could guarantee that the results were accurate, versus potentially getting an unexpected and confusing result.

1. the defaults are described in the signature - I can add them in the explanation too?
2. good idea, let me do that.
3 . will do.
4. ditto
5. I think it always uses the bloom filter for non int/long values
6. makes sense

1. The signature I saw was `ID_SET(columnName, 'sizeThresholdInBytes=1000;expectedInsertions=10000;fpp=0.03' )`, but I thought `expectedInsertions` default was 5M, and `sizeThreadholdInBytes` was 8MB. Oh, I see you edited these, cool.
2. Sorry, missed the `non` bit in the description in my fifth item.

I thought the values in the google doc were the defaults, but they weren't 'haha

<@U01ECR21TEE> We can add another optional argument to throw exception whenever the size exceeds the threshold, should be straight forward

Hi <@U01DS9A0DS4> - ok, looks good. I’d still love to see something about being able to use ids from a different table, as a way of doing a cross-table filter (except when using `IN__PARTITIONED__SUBQUERY`).

ok, lemme think of an example for that one

it would be great if `IN_SUBQUERY` could support use cases where your sub query has a group by / having clause: e.g.
```... WHERE IN_SUBQUERY(yearID, "SELECT yearID, count(*) from ... GROUP BY yearID HAVING yearID &gt;= 10) ...```

Hello, PR for some formatting changes and fixes for explain plan documentation: <https://github.com/pinot-contrib/pinot-docs/pull/77>. Would appreciate it if this could be merged. Thanks.

Fixed: <https://github.com/pinot-contrib/pinot-docs/pull/78>

Folks, please help review

<https://github.com/pinot-contrib/pinot-docs/pull/94|https://github.com/pinot-contrib/pinot-docs/pull/94>

Hello, this PR (<https://github.com/pinot-contrib/pinot-docs/pull/96>) is for adding more details to json queries document..

can I get help in approving this <https://github.com/pinot-contrib/pinot-docs/pull/97> ?

I don't seem to have write access on this repo

Hi, I've just send out a PR for an Explain plan doc update: <https://github.com/pinot-contrib/pinot-docs/pull/98>
The doc update is based on changes made in this PR: <https://github.com/apache/pinot/pull/8738>
can someone take a look. cc <@ULFH7G494>

Hello, can I please get a review and merge for <https://github.com/pinot-contrib/pinot-docs/pull/103>.  Thanks.

Can I get a review on this? Updating the docs to reflect that groovy is disabled by default now: <https://github.com/pinot-contrib/pinot-docs/pull/112>

This PR on Base64 transform func (<https://github.com/pinot-contrib/pinot-docs/pull/109>) has been merged but not is not shown on the user docs. Does anyone know what the issue might be? Thank you!

I’ve sync’ed up pinot docs from gitbook to github. Please note that the change for pinot-docs repo, should all go to `latest` branch which will reflect the `latest` version of pinot-docs.

Content of *<https://app.gitbook.com/o/-LtRX9NwSr7Ga7zA4piL/s/-LtH6nl58DdnZnelPdTc-887967055/|latest>* has been updated