Is regex_like function supported on map column? e....
# general
r
Is regex_like function supported on map column? e.g query
Copy code
SELECT mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES) FROM FeatureTest1 WHERE  REGEXP_LIKE(mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES), '^p.*')  LIMIT 5
Tried similar query on (pinot 0.7.1) and getting exception.
m
We are moving towards JSON data type, instead of MAP/List etc. For JSON data type, you can set a JSON index as well as a text index and do TEXT_MATCH queries. However, the text index is on the entire json blob treated as text, as opposed to individual fields.
j
It should work. What exception did you get?
r
@User Is there also plan to extend JSON_MATCH index to support regex kind of query on one of its key, value expression for value type is string. e.g https://github.com/apache/pinot/issues/6752
It should work. What exception did you get?
was getting below exception.
Copy code
[
  {
    "message": "QueryExecutionError:\njava.lang.IndexOutOfBoundsException\n\tat java.base/java.nio.Buffer.checkIndex(Unknown Source)\n\tat java.base/java.nio.DirectByteBuffer.get(Unknown Source)\n\tat org.apache.pinot.core.segment.memory.PinotByteBuffer.getByte(PinotByteBuffer.java:78)\n\tat org.apache.pinot.core.io.util.FixedByteValueReaderWriter.getUnpaddedString(FixedByteValueReaderWriter.java:59)\n\tat org.apache.pinot.core.segment.index.readers.BaseImmutableDictionary.getUnpaddedString(BaseImmutableDictionary.java:283)\n\tat org.apache.pinot.core.segment.index.readers.StringDictionary.getStringValue(StringDictionary.java:69)\n\tat org.apache.pinot.core.operator.filter.predicate.RegexpLikePredicateEvaluatorFactory$DictionaryBasedRegexpLikePredicateEvaluator.applySV(RegexpLikePredicateEvaluatorFactory.java:82)\n\tat org.apache.pinot.core.operator.dociditerators.ExpressionScanDocIdIterator.processProjectionBlock(ExpressionScanDocIdIterator.java:128)\n\tat org.apache.pinot.core.operator.dociditerators.ExpressionScanDocIdIterator.next(ExpressionScanDocIdIterator.java:81)\n\tat org.apache.pinot.core.operator.dociditerators.ExpressionScanDocIdIterator.advance(ExpressionScanDocIdIterator.java:106)\n\tat org.apache.pinot.core.operator.dociditerators.AndDocIdIterator.next(AndDocIdIterator.java:51)\n\tat org.apache.pinot.core.operator.DocIdSetOperator.getNextBlock(DocIdSetOperator.java:69)\n\tat org.apache.pinot.core.operator.DocIdSetOperator.getNextBlock(DocIdSetOperator.java:35)\n\tat org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49)",
    "errorCode": 200
  }
]
m
You can build both json and text index on the json column, and then perform regex match. But right now you can't do text index based on individual json fields.
r
Here regex match will be on entire json string, right?
@User Hey, this worked fine
SELECT mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES) FROM FeatureTest1 WHERE  REGEXP_LIKE(mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES), '^p.*')  LIMIT 5
once I added startTime and endTime filter on the time column and ``mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES) != ''`. Looks like there was an issue if either key or value is missing on some of the rows.
j
I think the problem might be
mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES)
will throw exception when
'k1'
does not exist in the
mapDim1__KEYS
. Can you try adding a filter
AND mapDim1__KEYS = 'k1'
and see if it works?