https://pinot.apache.org/ logo
Join Slack
Powered by
Team, I added new index and sortedColumn in the ta...
# troubleshooting
m
Team, I added new index and sortedColumn in the table config which was already ingesting data from Kafka stream. I used “AddTable” command to update the index. “jsonIndexColumns”: [ “entityMap” ], “sortedColumn”: [ “metric” ] I performed “Reload All Segments” in the UI. Is there any way to know if the indexing is complete?
k
check in the table page, reload status button
m
image.png
k
ah, not sure why its not supported for real-time table @Neha Pawar ^^
n
this was from a contributor in open source. He’s only done it for offline.
@Omkar Halikar is working on adding the realtime support
meanwhile, you can look at the status of indexing, bu going to the segment directory on the server instance, and looking at metadata.properties
m
@Neha Pawar Any specific field you would like me to look at?
image.png
I could not get JSON_MATCH expression to work in the query. I created a new table with the right indexes from the beginning. I dont get the errors while querying. The new table is small (2.8M records vs 92M records). My JSON_MATCH expression is still not fetching any result.
k
whats the query
m
Copy code
SELECT * FROM metric_v7 WHERE JSON_MATCH(entityMap, '"$.Db2Bufp"=''BP20''')
Copy code
Here is the table -

entityMap	                         metric	     metricTime
{"Db2Bufp":"BP20","Db2Ssid":"DT31"}	IDBBWRTI	1622701680000
{"Db2Bufp":"BP3","Db2Ssid":"DT31"}	IDBBPGWR	1622701680000
{"Db2Bufp":"BP0","Db2Ssid":"DT31"}	IDBBGTPG	1622701680000

Table config -

"tableIndexConfig": {
      "jsonIndexColumns": [
        "entityMap"
      ],
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [
        "metric"
      ],
      "loadMode": "MMAP",
k
@Jackie ^^
j
Which version of pinot are you running?
m
pinot:0.7.1
j
Can you try 
SELECT * FROM metric_v7 WHERE JSON_MATCH(entityMap, 'Db2Bufp=''BP20''')
and see if it works? We recently changed the syntax of the JSON_MATCH which is not included in 
0.7.1
👍 1
FYI, the sorted column config won't be picked up by the existing segments because that requires re-sorting the records, which cannot be done on server side
m
This query worked..
Ah.. As I was facing problems with the new index and sorted column, I started a new table from scratch
j
In that case, the new consumed segments should have the correct index generated
m
@Jackie I got the JSON index working. This is how my table looks now -
Copy code
entityMap	                         metric	     metricTime
{"Db2Bufp":"BP20","Db2Ssid":"DT31"}	IDBBWRTI	1622701680000
{"Db2Bufp":"BP3","Db2Ssid":"DT31"}	IDBBPGWR	1622701680000
{"Db2Bufp":"BP0","Db2Ssid":"DT31"}	IDBBGTPG	1622701680000

Table config -

"tableIndexConfig": {
      "jsonIndexColumns": [
        "entityMap"
....
{
          "columnName": "entityMap",
          "transformFunction": "jsonPathString(MetricSource, '$.entityMap', 'null')"
        },
The entityMap can be null sometimes. I wish to provide a default value but still continue to ingest the record. I am getting following exception -
Copy code
java.lang.IllegalStateException: Cannot flatten value node: null
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.
1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.spi.utils.JsonUtils.flatten(JsonUtils.java:246) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c
424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.realtime.impl.json.MutableJsonIndex.add(MutableJsonIndex.java:71) ~[pinot-all-0.7.1-jar-with-dependencies
.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:631) ~[pinot-all-0.7.1-jar-with
-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:475) ~[pinot-all-0.7.1-jar-with-dep
endencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:
497) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.
j
Hi, this issue is fixed in the latest master
In 0.7.1 you can put ‘{}’ (empty object) as the default value
👍 1
m
Tested with latest docker image. Confirmed, it works now.
Open in Slack
PreviousNext
Powered by