Hi @Seunghyun, thanks for your response. I have succesfully connected Superset to Pinot. My problem is knowing if it is posible to dedupe data in offline tables.

Do we know why duplication happened? Does your original data contain the duplicated rows? Pinot’s offline ingestion usually naively converts the input data, convert into Pinot segments, and load that into the cluster. Data is not expected to change.

Hi!!

new json = 1 row?

at this moment yes

then probably you may need the upsert feature

upsery is only for realtime tables ¿right?

so whenever there’s a data change, you can emit the change to kafka

can I emmit partial updates?

check

*Partial upserts*

section

👍

creating 1 pinot segment for each row is not a scalable model, we will end up having toooo many segments

yes, that is what I thought

👍

Hi Again. One more question (short one I hope). I managed to get things working with a realtime table and some kafka topics. Is working as expected but no files are written to deep storage (with offline tables there where files, so it seems to be properly configured). Is this the expected behaviour? Pinot will send data to deep storage when it considers?

By default the RT tables stay in memory, ref:

<https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime>

. For other behaviors, you might look into hybrid tables:

<https://docs.pinot.apache.org/basics/components/table#hybrid-table>

👍

• upsert is not working with the hybrid table so we need to keep the realtime tables only • for realtime, we first keep data in memory and flush data based on the threshold ◦ e.g.

"streamConfigs" : {
  "realtime.segment.flush.threshold.rows": "0",
  "realtime.segment.flush.threshold.time": "24h",
  "realtime.segment.flush.threshold.segment.size": "150M",
  "streamType": "kafka",
  "stream.kafka.consumer.type": "LowLevel",
  "stream.kafka.topic.name": "ClickStream",
  "stream.kafka.consumer.prop.auto.offset.reset" : "largest"
}

Hi again. I almost have reached my goals with Pinot, but there one las thing. I have a REALTIME table with a metric field. This realtime table ingest a Kafka topic and it has partial upserts cofigured. Every partial upsert works as expected excepting the metric field (an integer one configured as INCREMENT upsert. I have made queries to the the table with option(skipUpsert=true) to see the change history and i see that everytime the value at the metric column is 0, but i have sent many messages to the kafka topic geting partials upsert