https://pinot.apache.org/ logo
Join Slack
Powered by
a
pls help
these are the images i am using
I pulled latest pinot, still same error..
m
I think PrestoSQL/Trino may be more up to date than prestodb @Xiang Fu ?
a
i used the image from the document... Is there some other image i should use?
m
Oh then ignore my comment.
Let me check and get back
a
sure
Any luck @Mayank ? Or is there anything else I can try to circumvent this problem?
x
can you check if your pinot server config has
Copy code
pinot.server.instance.currentDataTableVersion=2
I think there was a recent pinot server upgrade, which make the internal data protocol version advanced than presto has
a
how can i check this ?
is it in the UI
?
x
for k8s
a
I am running in docker on my local
x
check
Copy code
kubectl get configmaps pinot-live-server-config -n pinot -o yaml
replace 
-n pinot
to your namespace
a
i am not running on kubernetes
x
if just docker
then I guess it doesn’t have
have do you start the pinot docker
a
Copy code
docker run \
    --network=pinot-demo \
    --name pinot-quickstart \
    -p 9000:9000 \
    -d apachepinot/pinot:latest QuickStart \
    -type hybrid
x
ic, then this one should have no such configs
can you try image 
apachepinot/pinot:0.7.1
a
ok sure..
Yeah with this the NPE is gone and i am able to query from presto... but the Query console on pinot UI is going blank..
is there any way to get the latest pinot working with this ? Not sure of what features i will miss by using 0.7.0..
getting this blank query screen.
x
hmm, can you try to clean the cache or try in another browser?
a
Yeah just tried that..
it worked in incognito..
đź‘Ť 1
I will try the join on presto now!! Thank you.
x
đź‘Ť
a
ignore the last message.. i will investigate more and get back.
x
Can you try this
Copy code
SET SESSION pinot.limit_larger_for_segment=200000000; SELECT ...
I thought default pinot.limit_larger_for_segment should be 2147483647
a
let me try
that didn't make any change.. actually the join is doing something strange.. its not giving the correct result..
Copy code
presto:default> select txName,sum(a.amount) from txn a left join txtypes b on a.txtype=b.txtype group by txName;
 txName  |       _col1       
---------+-------------------
 Invoice | 2453.240119934082 
(1 row)

Query 20210512_173300_00076_imcrr, FINISHED, 1 node
Splits: 100 total, 100 done (100.00%)
0:00 [0 rows, 62B] [0 rows/s, 157B/s]

presto:default> select txName,sa from (select txtype,sum(amount) sa from txn group by txtype) a join txtypes b on a.txtype=b.txtype;
 txName  |         sa         
---------+--------------------
 Invoice | 2539.4801235198975 
(1 row)

Query 20210512_173312_00077_imcrr, FINISHED, 1 node
Splits: 67 total, 67 done (100.00%)
0:00 [0 rows, 42B] [0 rows/s, 210B/s]
the second query 's result is correct...
First query is ignoring 2 rows..
x
hmm, how large is table txn
a
its a very small table.. just 4 records in txn and 2 records in txtypes..
x
then try to do join all and see?
Copy code
select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype
a
Copy code
presto:default> select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype ;
 txName  |       amount       | txtype 
---------+--------------------+--------
 Invoice |    2342.1201171875 |      1 
 Invoice | 111.12000274658203 |      1 
(2 rows)

Query 20210512_173837_00085_imcrr, FINISHED, 1 node
Splits: 68 total, 68 done (100.00%)
0:00 [0 rows, 62B] [0 rows/s, 244B/s]

presto:default> select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype limit 100;
 txName  | amount  | txtype 
---------+---------+--------
 Invoice | 2342.12 |      1 
 Invoice |   65.12 |      1 
 Invoice |   21.12 |      1 
 Invoice |  111.12 |      1 
(4 rows)
without limit it shows 2 records..
with limit , it shows all 4
x
hmm
what’s generated pinot query for without limit case
a
Copy code
GeneratedPinotQuery{query=SELECT amount, txtype FROM txn__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1, format=SQL, table=txn, expectedColumnIndices=[], groupByClauses=0,
x
can you check presto config in your docker container log
a
which config?
x
what’s the config for
Copy code
pinot.limit-large-for-segment
you can search log for this
I think it should be set as 1
a
not able to find this in the docker logs..
x
for quickstart we set this to 1
Copy code
connector.name=pinot
pinot.controller-urls=pinot-quickstart:9000
pinot.controller-rest-service=pinot-quickstart:9000

pinot.limit-large-for-segment=1
pinot.allow-multiple-aggregations=true
pinot.use-date-trunc=true
pinot.infer-date-type-in-schema=true
pinot.infer-timestamp-type-in-schema=true
so it’s intentional
a
ok got it.. so is this something I can reset ?
x
you can try to create this 
pinot_quickstart.properties
file
then mount it to docker container
also the session config should work
a
ok i will try the session cofig first..
actually i had set earlier when you pointed.
Copy code
SET SESSION pinot.limit_larger_for_segment=200000000;
but that didnt work
x
SET SESSION pinot.limit_larger_for_segment=2147483647;
try this
then send the query again
a
ok
x
you can check the explain for that
a
didnt work.. i guess the session setthing is not getting picked up..
x
hmm
which presto image are you using
a
Copy code
243aa15aff9d
x
so explain doesn’t give the right limit?
a
correct that still has limit 1
x
I tried on my side, it gives me the
Copy code
GeneratedPinotQuery{query=SELECT DestStateName FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1000,
when I do
Copy code
presto:default> SET SESSION pinot.limit_larger_for_segment=1000;
hmm
a
hmm .. not sure whats wrong.. are you using the same image?
x
I think as long as you can set the session config
then it should be fine
a
ok.. another thing,.. I tried this as well...
Copy code
presto:default> select * from txn;
       amount       |      id       | txtype | tenant |           ts            
--------------------+---------------+--------+--------+-------------------------
 111.12000274658203 | 101_1_2020102 |      1 |    101 | 2021-05-12 12:45:20.744 
    2342.1201171875 | 101_1_2020105 |      1 |    101 | 2021-05-12 12:44:50.744 
(2 rows)
i should have gotten 5 records..
but only got 2..
but i tried select * from airlineStats, that is giving me multiple records...
x
hmm
which catalog are you using
what’s the explain on this query
a
Copy code
uery=SELECT AirTime FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1
Copy code
PinotQuery{query=SELECT AirTime FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1, format=SQL, table=airlineStats, expectedColumnIn
x
I think it’s getting 1 record per segment
then merge
but this session config doesn’t work really interesting
can you try presto image:
Copy code
apachepinot/pinot-presto:0.254-SNAPSHOT-54a7ec79a3-20210512
a
yeah something strange..
yeah i can try
same result ..
i set the session as well..
x
hmm
then change the config file and mount it
a
ok.. in production, this value how will it be decided?
i work out of india time zone.. so i will test this config file thing tomorrow morning my time and update here..
x
yes
in prod, you anyway need to have your own config file and set it accordingly
a
is there anything in the docs that talks about this? I would like to understand how to set it up..
x
for prod, usually people go with kubernetes and helmChart, you can find that on docs.pinot.apache.org
also can you give it a try for this image: 
apachepinot/pinot-presto:0.254-SNAPSHOT-f434ea10a5-20210513
I made the image to use default configs
Open in Slack
PreviousNext
Powered by