Hello everyone! I'm writing an article for medium of a real case implementation of Apache Pinot as a way of studying. For that, I'm using a Strimzi cluster and the Twitter Api Kafka Connector (that you can find at https://www.confluent.io/hub/jcustenborder/kafka-connect-twitter) running on Minikube, to get data from twitter's api and ingest it into Pinot. I followed the steps explained in this video making some adaptations to run on kubernetes. That way I was able to infer the schema of the "twitter-sample.json" file attached to this message by generating the schema file "twitter-old-schema.json", after that I had to remove some fields: "schema.type", "schema.fields", "schema.optional", "schema.name", and remove the prefix "payload." of every column to generate the file "twitter-schema.json". Then with this schema file and with the table config file "twitter-config.json" I created the REALTIME table "twitter-status-events" (using the column "CreatedAt" as datetime column) using pinot-admin.sh inside pinot controller's pod. But for some reason that I don't know I'm not getting any record in this table, do you have any idea what I'm doing wrong ? (more information replied to this comment)

Hi @Marlon Félix, let me take a look at your setup, will get back.

thank you!

Can you share some sample input records, or is that

twitter-sample.json

?

"twitter-sample.json" is a sample record

Hey @Marlon Félix, seems like you are assuming that you can ingest complex structs using

dot

notation. This is not the case at the moment. For example, you need to extract the nested

User.id

from input to write to a column in pinot.

I understood. I saw in this video there was this possibility when adding the parameter

"ingestionConfig": {
      "complexTypeConfig": {
        "delimiter": "."
      }

To the table config file.

Ah yes. Did you specify transform functions though?

I didn't. Okay, thank you! I will try it and share the results.

👍