hello team We have a JSON file with 200 attributes...
# general
s
hello team We have a JSON file with 200 attributes, structured as simple key-value pairs (no nested or complex JSON). We are considering using Pinot's transformation configuration to flatten these attributes, creating each attribute name as a separate column. For example, we receive the following JSON in a Kafka topic:
Copy code
json

{
  "countryname": "USA",
  
  "attributes": { "col1": "val1", "col2": "val2", ... }
}
We plan to transform and store it in Pinot as: country col1 col2 col3...col100 USA val1 val2... Questions: 1. Will this transformation approach cause any performance impact in production? 2. Would it be better to write our own transformation tool instead of relying on Pinot's transformation configuration?
n
you should be able to use pinot's transformation out of the box for this https://docs.pinot.apache.org/basics/data-import/pinot-input-formats/complex-type-examples
the approach shouldn't cause any performance impact.
curious, why are you preferring to transform it into columns, instead of storing as json and using json index?
s
Any example on json index to retrive the column name and value