Channels
#troubleshooting
Title
a
Ambika
05/12/2021, 10:37 AMpls help
these are the images i am using
I pulled latest pinot, still same error..
m
Mayank
05/12/2021, 3:33 PM
I think PrestoSQL/Trino may be more up to date than prestodb @Xiang Fu ?
a
Ambika
05/12/2021, 3:34 PMi used the image from the document... Is there some other image i should use?
m
Mayank
05/12/2021, 3:35 PM
Oh then ignore my comment.
Let me check and get back
a
Ambika
05/12/2021, 3:35 PMsure
Any luck @Mayank ? Or is there anything else I can try to circumvent this problem?
x
Xiang Fu
05/12/2021, 5:18 PM
can you check if your pinot server config has
Copy code
pinot.server.instance.currentDataTableVersion=2I think there was a recent pinot server upgrade, which make the internal data protocol version advanced than presto has
a
Ambika
05/12/2021, 5:19 PMhow can i check this ?
is it in the UI
?
x
Xiang Fu
05/12/2021, 5:19 PM
for k8s
a
Ambika
05/12/2021, 5:19 PMI am running in docker on my local
x
Xiang Fu
05/12/2021, 5:20 PM
check
Copy code
kubectl get configmaps pinot-live-server-config -n pinot -o yamlreplace
-n pinota
Ambika
05/12/2021, 5:20 PMi am not running on kubernetes
x
Xiang Fu
05/12/2021, 5:21 PM
if just docker
then I guess it doesn’t have
have do you start the pinot docker
a
Ambika
05/12/2021, 5:21 PM Copy code
docker run \
--network=pinot-demo \
--name pinot-quickstart \
-p 9000:9000 \
-d apachepinot/pinot:latest QuickStart \
-type hybridx
Xiang Fu
05/12/2021, 5:22 PM
ic, then this one should have no such configs
can you try image
apachepinot/pinot:0.7.1a
Ambika
05/12/2021, 5:24 PMok sure..
Yeah with this the NPE is gone and i am able to query from presto... but the Query console on pinot UI is going blank..
is there any way to get the latest pinot working with this ? Not sure of what features i will miss by using 0.7.0..
getting this blank query screen.
x
Xiang Fu
05/12/2021, 5:43 PM
hmm, can you try to clean the cache or try in another browser?
a
Ambika
05/12/2021, 5:44 PMYeah just tried that..
it worked in incognito..
👍 1
I will try the join on presto now!! Thank you.
x
Xiang Fu
05/12/2021, 5:45 PM
👍
a
Ambika
05/12/2021, 6:15 PMignore the last message.. i will investigate more and get back.
x
Xiang Fu
05/12/2021, 6:16 PM
Can you try this
Copy code
SET SESSION pinot.limit_larger_for_segment=200000000; SELECT ...I thought default pinot.limit_larger_for_segment should be 2147483647
a
Ambika
05/12/2021, 6:17 PMlet me try
that didn't make any change.. actually the join is doing something strange.. its not giving the correct result..
Copy code
presto:default> select txName,sum(a.amount) from txn a left join txtypes b on a.txtype=b.txtype group by txName;
txName | _col1
---------+-------------------
Invoice | 2453.240119934082
(1 row)
Query 20210512_173300_00076_imcrr, FINISHED, 1 node
Splits: 100 total, 100 done (100.00%)
0:00 [0 rows, 62B] [0 rows/s, 157B/s]
presto:default> select txName,sa from (select txtype,sum(amount) sa from txn group by txtype) a join txtypes b on a.txtype=b.txtype;
txName | sa
---------+--------------------
Invoice | 2539.4801235198975
(1 row)
Query 20210512_173312_00077_imcrr, FINISHED, 1 node
Splits: 67 total, 67 done (100.00%)
0:00 [0 rows, 42B] [0 rows/s, 210B/s]the second query 's result is correct...
First query is ignoring 2 rows..
x
Xiang Fu
05/12/2021, 6:24 PM
hmm, how large is table txn
a
Ambika
05/12/2021, 6:26 PMits a very small table.. just 4 records in txn and 2 records in txtypes..
x
Xiang Fu
05/12/2021, 6:26 PM
then try to do join all and see?
Copy code
select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtypea
Ambika
05/12/2021, 6:27 PM Copy code
presto:default> select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype ;
txName | amount | txtype
---------+--------------------+--------
Invoice | 2342.1201171875 | 1
Invoice | 111.12000274658203 | 1
(2 rows)
Query 20210512_173837_00085_imcrr, FINISHED, 1 node
Splits: 68 total, 68 done (100.00%)
0:00 [0 rows, 62B] [0 rows/s, 244B/s]
presto:default> select txName, a.amount, a.txtype from txn a left join txtypes b on a.txtype=b.txtype limit 100;
txName | amount | txtype
---------+---------+--------
Invoice | 2342.12 | 1
Invoice | 65.12 | 1
Invoice | 21.12 | 1
Invoice | 111.12 | 1
(4 rows)without limit it shows 2 records..
with limit , it shows all 4
x
Xiang Fu
05/12/2021, 6:28 PM
hmm
what’s generated pinot query for without limit case
a
Ambika
05/12/2021, 6:29 PM Copy code
GeneratedPinotQuery{query=SELECT amount, txtype FROM txn__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1, format=SQL, table=txn, expectedColumnIndices=[], groupByClauses=0,x
Xiang Fu
05/12/2021, 6:40 PM
can you check presto config in your docker container log
a
Ambika
05/12/2021, 6:40 PMwhich config?
x
Xiang Fu
05/12/2021, 6:41 PM
what’s the config for
Copy code
pinot.limit-large-for-segmentyou can search log for this
I think it should be set as 1
a
Ambika
05/12/2021, 6:44 PMnot able to find this in the docker logs..
x
Xiang Fu
05/12/2021, 6:44 PM
for quickstart we set this to 1
Copy code
connector.name=pinot
pinot.controller-urls=pinot-quickstart:9000
pinot.controller-rest-service=pinot-quickstart:9000
pinot.limit-large-for-segment=1
pinot.allow-multiple-aggregations=true
pinot.use-date-trunc=true
pinot.infer-date-type-in-schema=true
pinot.infer-timestamp-type-in-schema=trueso it’s intentional
a
Ambika
05/12/2021, 6:44 PMok got it.. so is this something I can reset ?
x
Xiang Fu
05/12/2021, 6:45 PM
you can try to create this
pinot_quickstart.propertiesthen mount it to docker container
also the session config should work
a
Ambika
05/12/2021, 6:46 PMok i will try the session cofig first..
actually i had set earlier when you pointed.
Copy code
SET SESSION pinot.limit_larger_for_segment=200000000;but that didnt work
x
Xiang Fu
05/12/2021, 6:47 PM
SET SESSION pinot.limit_larger_for_segment=2147483647;
try this
then send the query again
a
Ambika
05/12/2021, 6:48 PMok
x
Xiang Fu
05/12/2021, 6:49 PM
you can check the explain for that
a
Ambika
05/12/2021, 6:49 PMdidnt work.. i guess the session setthing is not getting picked up..
x
Xiang Fu
05/12/2021, 6:49 PM
hmm
which presto image are you using
a
Ambika
05/12/2021, 6:49 PM Copy code
243aa15aff9dx
Xiang Fu
05/12/2021, 6:50 PM
so explain doesn’t give the right limit?
a
Ambika
05/12/2021, 6:50 PMcorrect that still has limit 1
x
Xiang Fu
05/12/2021, 6:50 PM
I tried on my side, it gives me the
Copy code
GeneratedPinotQuery{query=SELECT DestStateName FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1000,when I do
Copy code
presto:default> SET SESSION pinot.limit_larger_for_segment=1000;hmm
a
Ambika
05/12/2021, 6:52 PMhmm .. not sure whats wrong.. are you using the same image?
x
Xiang Fu
05/12/2021, 6:54 PM
I think as long as you can set the session config
then it should be fine
a
Ambika
05/12/2021, 6:57 PMok.. another thing,.. I tried this as well...
Copy code
presto:default> select * from txn;
amount | id | txtype | tenant | ts
--------------------+---------------+--------+--------+-------------------------
111.12000274658203 | 101_1_2020102 | 1 | 101 | 2021-05-12 12:45:20.744
2342.1201171875 | 101_1_2020105 | 1 | 101 | 2021-05-12 12:44:50.744
(2 rows)i should have gotten 5 records..
but only got 2..
but i tried select * from airlineStats, that is giving me multiple records...
x
Xiang Fu
05/12/2021, 6:57 PM
hmm
which catalog are you using
what’s the explain on this query
a
Ambika
05/12/2021, 6:59 PM Copy code
uery=SELECT AirTime FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1 Copy code
PinotQuery{query=SELECT AirTime FROM airlineStats__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 1, format=SQL, table=airlineStats, expectedColumnInx
Xiang Fu
05/12/2021, 6:59 PM
I think it’s getting 1 record per segment
then merge
but this session config doesn’t work really interesting
can you try presto image:
Copy code
apachepinot/pinot-presto:0.254-SNAPSHOT-54a7ec79a3-20210512a
Ambika
05/12/2021, 7:00 PMyeah something strange..
yeah i can try
same result ..
i set the session as well..
x
Xiang Fu
05/12/2021, 7:05 PM
hmm
then change the config file and mount it
a
Ambika
05/12/2021, 7:06 PMok.. in production, this value how will it be decided?
i work out of india time zone.. so i will test this config file thing tomorrow morning my time and update here..
x
Xiang Fu
05/12/2021, 7:07 PM
yes
in prod, you anyway need to have your own config file and set it accordingly
a
Ambika
05/12/2021, 7:09 PMis there anything in the docs that talks about this? I would like to understand how to set it up..
x
Xiang Fu
05/13/2021, 8:46 PM
for prod, usually people go with kubernetes and helmChart, you can find that on docs.pinot.apache.org
also can you give it a try for this image:
apachepinot/pinot-presto:0.254-SNAPSHOT-f434ea10a5-20210513I made the image to use default configs