https://pinot.apache.org/ logo
Channels
Powered by
m

Machhindra

06/02/2021, 8:03 PM
Team, I added new index and sortedColumn in the table config which was already ingesting data from Kafka stream. I used “AddTable” command to update the index. “jsonIndexColumns”: [ “entityMap” ], “sortedColumn”: [ “metric” ] I performed “Reload All Segments” in the UI. Is there any way to know if the indexing is complete?
k

Kishore G

06/02/2021, 8:04 PM
check in the table page, reload status button
m

Machhindra

06/02/2021, 8:05 PM
message has been deleted
k

Kishore G

06/02/2021, 9:59 PM
ah, not sure why its not supported for real-time table @Neha Pawar ^^
n

Neha Pawar

06/02/2021, 10:01 PM
this was from a contributor in open source. He’s only done it for offline.
@Omkar Halikar is working on adding the realtime support
meanwhile, you can look at the status of indexing, bu going to the segment directory on the server instance, and looking at metadata.properties
m

Machhindra

06/03/2021, 2:28 PM
@Neha Pawar Any specific field you would like me to look at?
message has been deleted
I could not get JSON_MATCH expression to work in the query. I created a new table with the right indexes from the beginning. I dont get the errors while querying. The new table is small (2.8M records vs 92M records). My JSON_MATCH expression is still not fetching any result.
k

Kishore G

06/03/2021, 5:33 PM
whats the query
m

Machhindra

06/03/2021, 5:35 PM
Copy code
SELECT * FROM metric_v7 WHERE JSON_MATCH(entityMap, '"$.Db2Bufp"=''BP20''')
Copy code
Here is the table -

entityMap	                         metric	     metricTime
{"Db2Bufp":"BP20","Db2Ssid":"DT31"}	IDBBWRTI	1622701680000
{"Db2Bufp":"BP3","Db2Ssid":"DT31"}	IDBBPGWR	1622701680000
{"Db2Bufp":"BP0","Db2Ssid":"DT31"}	IDBBGTPG	1622701680000

Table config -

"tableIndexConfig": {
      "jsonIndexColumns": [
        "entityMap"
      ],
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [
        "metric"
      ],
      "loadMode": "MMAP",
k

Kishore G

06/03/2021, 5:42 PM
@Jackie ^^
j

Jackie

06/03/2021, 5:43 PM
Which version of pinot are you running?
m

Machhindra

06/03/2021, 5:44 PM
pinot:0.7.1
j

Jackie

06/03/2021, 5:44 PM
Can you try 
SELECT * FROM metric_v7 WHERE JSON_MATCH(entityMap, 'Db2Bufp=''BP20''')
and see if it works? We recently changed the syntax of the JSON_MATCH which is not included in 
0.7.1
👍 1
FYI, the sorted column config won't be picked up by the existing segments because that requires re-sorting the records, which cannot be done on server side
m

Machhindra

06/03/2021, 5:46 PM
This query worked..
Ah.. As I was facing problems with the new index and sorted column, I started a new table from scratch
j

Jackie

06/03/2021, 5:48 PM
In that case, the new consumed segments should have the correct index generated
m

Machhindra

06/09/2021, 6:22 AM
@Jackie I got the JSON index working. This is how my table looks now -
Copy code
entityMap	                         metric	     metricTime
{"Db2Bufp":"BP20","Db2Ssid":"DT31"}	IDBBWRTI	1622701680000
{"Db2Bufp":"BP3","Db2Ssid":"DT31"}	IDBBPGWR	1622701680000
{"Db2Bufp":"BP0","Db2Ssid":"DT31"}	IDBBGTPG	1622701680000

Table config -

"tableIndexConfig": {
      "jsonIndexColumns": [
        "entityMap"
....
{
          "columnName": "entityMap",
          "transformFunction": "jsonPathString(MetricSource, '$.entityMap', 'null')"
        },
The entityMap can be null sometimes. I wish to provide a default value but still continue to ingest the record. I am getting following exception -
Copy code
java.lang.IllegalStateException: Cannot flatten value node: null
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.
1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.spi.utils.JsonUtils.flatten(JsonUtils.java:246) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c
424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.realtime.impl.json.MutableJsonIndex.add(MutableJsonIndex.java:71) ~[pinot-all-0.7.1-jar-with-dependencies
.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:631) ~[pinot-all-0.7.1-jar-with
-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:475) ~[pinot-all-0.7.1-jar-with-dep
endencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:
497) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.
j

Jackie

06/09/2021, 7:25 AM
Hi, this issue is fixed in the latest master
In 0.7.1 you can put ‘{}’ (empty object) as the default value
👍 1
m

Machhindra

06/09/2021, 1:11 PM
Tested with latest docker image. Confirmed, it works now.
Powered by