I want to define the

mode

columns as

$.properties.mode

, which is just simple json flattening, From the docs, I was not able to understand the correct syntax for

jsonPathString

to use in the tableConfig.

And the

request_failure

column is a derived column based on

$.properties.status

. I got to know after reading docs that chaining transformations isn’t supported in pinot, So I can’t define a column

status = $.properties.status_code

and then use it to define the other columns as

request_failure = if(status == 'created', 1, 0)

. So I think, I need to write a groovy script to extract the value from nested json and apply the if/else logic. But to extract a value from nested json, I would have to import json slurper in groovy(not so familiar with groovy, but this is what I found on SOF/internet to parse json in groovy). So my question here is, does pinot support import statements in the groovy script? If not, how can I achieve this transformation in pinot?

To give context around the use case, this use-case is currently onboarded to druid and then alerts are setup in thirdeye on top of the datasource in druid. We were facing issues with query response time because of one column. So we are trying to migrate the datasource to pinot and then try out star tree indexing with a hope that we would see much better performance. As of now, we are facing issues in migrating the datasource from druid to pinot(All the required transformations are supported in druid).

Do let me know, if more info is required.

We are running the image with

latest

tag.

And also I am trying to apply

filterConfig

, but it isn’t working. Here is my tableConfig

{
  "tableName": "x_fts_merchant_events_REALTIME",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "schemaName": "x_fts_merchant_events_dimensions",
    "timeColumnName": "event_timestamp",
    "timeType": "MILLISECONDS",
    "replicasPerPartition": "1",
    "retentionTimeValue": "1",
    "retentionTimeUnit": "DAYS",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "tableIndexConfig": {
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "LowLevel",
      "stream.kafka.topic.name": "x-fts-events-kafka",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "kafka-kafka-bootstrap.kafka.svc.cluster.local:9092",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    },
    "loadMode": "MMAP"
  },
  "metadata": {},
  "ingestionConfig": {
    "filterConfig": {
      "filterFunction": "Groovy({event_name == 'abc'}, event_name)"
    }
  }
}

this demo also uses it:

filter config looks right, it should work

why do you think it’s not working? are you still seeing event_name ‘abc’ in the ingested data?

are you still seeing event_name ‘abc’ in the ingested data?

Not able to see this. Ideally, I should be able to.

Chaining is supported now

Just to confirm. I first extract the field from json using inbuilt function, then I can use groovy to derive a field from the extracted column, correct?