https://pinot.apache.org/ logo
a

Ambika

05/12/2021, 10:37 AM
pls help
these are the images i am using
I pulled latest pinot, still same error..
m

Mayank

05/12/2021, 3:33 PM
I think PrestoSQL/Trino may be more up to date than prestodb @Xiang Fu ?
a

Ambika

05/12/2021, 3:34 PM
i used the image from the document... Is there some other image i should use?
m

Mayank

05/12/2021, 3:35 PM
Oh then ignore my comment.
Let me check and get back
a

Ambika

05/12/2021, 3:35 PM
sure
Any luck @Mayank ? Or is there anything else I can try to circumvent this problem?
x

Xiang Fu

05/12/2021, 5:18 PM
can you check if your pinot server config has
Copy code
pinot.server.instance.currentDataTableVersion=2
I think there was a recent pinot server upgrade, which make the internal data protocol version advanced than presto has
a

Ambika

05/12/2021, 5:19 PM
how can i check this ?
is it in the UI
?
x

Xiang Fu

05/12/2021, 5:19 PM
for k8s
a

Ambika

05/12/2021, 5:19 PM
I am running in docker on my local
x

Xiang Fu

05/12/2021, 5:20 PM
check
Copy code
kubectl get configmaps pinot-live-server-config -n pinot -o yaml
replace
-n pinot
to your namespace
a

Ambika

05/12/2021, 5:20 PM
i am not running on kubernetes
x

Xiang Fu

05/12/2021, 5:21 PM
if just docker
then I guess it doesn’t have
have do you start the pinot docker
a

Ambika

05/12/2021, 5:21 PM
Copy code
docker run \
    --network=pinot-demo \
    --name pinot-quickstart \
    -p 9000:9000 \
    -d apachepinot/pinot:latest QuickStart \
    -type hybrid
x

Xiang Fu

05/12/2021, 5:22 PM
ic, then this one should have no such configs
can you try image
apachepinot/pinot:0.7.1
a

Ambika

05/12/2021, 5:24 PM
ok sure..
Yeah with this the NPE is gone and i am able to query from presto... but the Query console on pinot UI is going blank..
is there any way to get the latest pinot working with this ? Not sure of what features i will miss by using 0.7.0..
getting this blank query screen.
x

Xiang Fu

05/12/2021, 5:43 PM
hmm, can you try to clean the cache or try in another browser?
a

Ambika

05/12/2021, 5:44 PM
Yeah just tried that..
it worked in incognito..
👍 1
I will try the join on presto now!! Thank you.
x

Xiang Fu

05/12/2021, 5:45 PM
👍
a

Ambika

05/12/2021, 6:15 PM
ignore the last message.. i will investigate more and get back.
x

Xiang Fu

05/12/2021, 6:16 PM
Can you try this
Copy code
SET SESSION pinot.limit_larger_for_segment=200000000; SELECT ...
I thought default pinot.limit_larger_for_segment should be 2147483647
a

Ambika

05/12/2021, 6:17 PM
let me try
that didn't make any change.. actually the join is doing something strange.. its not giving the correct result..
Copy code
presto:default> select txName,sum(a.amount) from txn a left join txtypes b on a.txtype=b.txtype group by txName;
 txName  |       _col1       
---------+-------------------
 Invoice | 2453.240119934082 
(1 row)

Query 20210512_173300_00076_imcrr, FINISHED, 1 node
Splits: 100 total, 100 done (100.00%)
0:00 [0 rows, 62B] [0 rows/s, 157B/s]

presto:default> select txName,sa from (select txtype,sum(amount) sa from txn group by txtype) a join txtypes b on a.txtype=b.txtype;
 txName  |         sa         
---------+--------------------
 Invoice | 2539.4801235198975 
(1 row)

Query 20210512_173312_00077_imcrr, FINISHED, 1 node
Splits: 67 total, 67 done (100.00%)
0:00 [0 rows, 42B] [0 rows/s, 210B/s]
the second query 's result is correct...
First query is ignoring 2 rows..
x

Xiang Fu

05/12/2021, 6:24 PM
hmm, how large is table txn
a

Ambika

05/12/2021, 6:26 PM
its a very small table.. just 4 records in txn and 2 records in txtypes..
x

Xiang Fu

05/12/2021, 6:26 PM
then try to do join all and see?
Copy code
select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype
a

Ambika

05/12/2021, 6:27 PM
Copy code
presto:default> select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype ;
 txName  |       amount       | txtype 
---------+--------------------+--------
 Invoice |    2342.1201171875 |      1 
 Invoice | 111.12000274658203 |      1 
(2 rows)

Query 20210512_173837_00085_imcrr, FINISHED, 1 node
Splits: 68 total, 68 done (100.00%)
0:00 [0 rows, 62B] [0 rows/s, 244B/s]

presto:default> select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype limit 100;
 txName  | amount  | txtype 
---------+---------+--------
 Invoice | 2342.12 |      1 
 Invoice |   65.12 |      1 
 Invoice |   21.12 |      1 
 Invoice |  111.12 |      1 
(4 rows)
without limit it shows 2 records..
with limit , it shows all 4
x

Xiang Fu

05/12/2021, 6:28 PM
hmm
what’s generated pinot query for without limit case
a

Ambika

05/12/2021, 6:29 PM
Copy code
GeneratedPinotQuery{query=SELECT amount, txtype FROM txn__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1, format=SQL, table=txn, expectedColumnIndices=[], groupByClauses=0,
x

Xiang Fu

05/12/2021, 6:40 PM
can you check presto config in your docker container log
a

Ambika

05/12/2021, 6:40 PM
which config?
x

Xiang Fu

05/12/2021, 6:41 PM
what’s the config for
Copy code
pinot.limit-large-for-segment
you can search log for this
I think it should be set as 1
a

Ambika

05/12/2021, 6:44 PM
not able to find this in the docker logs..
x

Xiang Fu

05/12/2021, 6:44 PM
for quickstart we set this to 1
Copy code
connector.name=pinot
pinot.controller-urls=pinot-quickstart:9000
pinot.controller-rest-service=pinot-quickstart:9000

pinot.limit-large-for-segment=1
pinot.allow-multiple-aggregations=true
pinot.use-date-trunc=true
pinot.infer-date-type-in-schema=true
pinot.infer-timestamp-type-in-schema=true
so it’s intentional
a

Ambika

05/12/2021, 6:44 PM
ok got it.. so is this something I can reset ?
x

Xiang Fu

05/12/2021, 6:45 PM
you can try to create this
pinot_quickstart.properties
file
then mount it to docker container
also the session config should work
a

Ambika

05/12/2021, 6:46 PM
ok i will try the session cofig first..
actually i had set earlier when you pointed.
Copy code
SET SESSION pinot.limit_larger_for_segment=200000000;
but that didnt work
x

Xiang Fu

05/12/2021, 6:47 PM
SET SESSION pinot.limit_larger_for_segment=2147483647;
try this
then send the query again
a

Ambika

05/12/2021, 6:48 PM
ok
x

Xiang Fu

05/12/2021, 6:49 PM
you can check the explain for that
a

Ambika

05/12/2021, 6:49 PM
didnt work.. i guess the session setthing is not getting picked up..
x

Xiang Fu

05/12/2021, 6:49 PM
hmm
which presto image are you using
a

Ambika

05/12/2021, 6:49 PM
Copy code
243aa15aff9d
x

Xiang Fu

05/12/2021, 6:50 PM
so explain doesn’t give the right limit?
a

Ambika

05/12/2021, 6:50 PM
correct that still has limit 1
x

Xiang Fu

05/12/2021, 6:50 PM
I tried on my side, it gives me the
Copy code
GeneratedPinotQuery{query=SELECT DestStateName FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1000,
when I do
Copy code
presto:default> SET SESSION pinot.limit_larger_for_segment=1000;
hmm
a

Ambika

05/12/2021, 6:52 PM
hmm .. not sure whats wrong.. are you using the same image?
x

Xiang Fu

05/12/2021, 6:54 PM
I think as long as you can set the session config
then it should be fine
a

Ambika

05/12/2021, 6:57 PM
ok.. another thing,.. I tried this as well...
Copy code
presto:default> select * from txn;
       amount       |      id       | txtype | tenant |           ts            
--------------------+---------------+--------+--------+-------------------------
 111.12000274658203 | 101_1_2020102 |      1 |    101 | 2021-05-12 12:45:20.744 
    2342.1201171875 | 101_1_2020105 |      1 |    101 | 2021-05-12 12:44:50.744 
(2 rows)
i should have gotten 5 records..
but only got 2..
but i tried select * from airlineStats, that is giving me multiple records...
x

Xiang Fu

05/12/2021, 6:57 PM
hmm
which catalog are you using
what’s the explain on this query
a

Ambika

05/12/2021, 6:59 PM
Copy code
uery=SELECT AirTime FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1
Copy code
PinotQuery{query=SELECT AirTime FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1, format=SQL, table=airlineStats, expectedColumnIn
x

Xiang Fu

05/12/2021, 6:59 PM
I think it’s getting 1 record per segment
then merge
but this session config doesn’t work really interesting
can you try presto image:
Copy code
apachepinot/pinot-presto:0.254-SNAPSHOT-54a7ec79a3-20210512
a

Ambika

05/12/2021, 7:00 PM
yeah something strange..
yeah i can try
same result ..
i set the session as well..
x

Xiang Fu

05/12/2021, 7:05 PM
hmm
then change the config file and mount it
a

Ambika

05/12/2021, 7:06 PM
ok.. in production, this value how will it be decided?
i work out of india time zone.. so i will test this config file thing tomorrow morning my time and update here..
x

Xiang Fu

05/12/2021, 7:07 PM
yes
in prod, you anyway need to have your own config file and set it accordingly
a

Ambika

05/12/2021, 7:09 PM
is there anything in the docs that talks about this? I would like to understand how to set it up..
x

Xiang Fu

05/13/2021, 8:46 PM
for prod, usually people go with kubernetes and helmChart, you can find that on docs.pinot.apache.org
also can you give it a try for this image:
apachepinot/pinot-presto:0.254-SNAPSHOT-f434ea10a5-20210513
I made the image to use default configs