Hi Team, I use Apache Pinot 1.4.0 for explorational analysis of raw events (small time windows) and for upserts based on primary key. The raw events are flat json format (~3.000 Events/second). The problem is that the attribute names are containing "-" like "event-timestamp" or "event-id". I guess Pinot does not like "-" in attribute names. My first idea was to use Apache Nifi for transformation and replace "-" with "_". Nifi seems not to be the right solution for this use case. What do you think would be an efficient solution (3.000-5.000Events/second)? Thank you :)

Have you tried transform functions in Pinot?

see if ingestion transform function like below works for you or not:

{
  "tableName": "events",
  "tableType": "REALTIME",
  "ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "event_id",
        "transformFunction": "\"event-id\""
      },
      {
        "columnName": "event_timestamp",
        "transformFunction": "\"event-timestamp\""
      },
      {
        "columnName": "user_id",
        "transformFunction": "\"user-id\""
      }
    ]
  }
}

It works for me, thank you very much! Table ingestion rate seems to be stable too.

Nice!