Hey all! We use relational database as the source of truth and we are looking for options to replicate the data to Pinot as real-time as possible. We may need some transformations in-between. We don't have a "large" dataset, mostly under 1TB. What options do we have? As a side note we want to avoid CDC.

Any reason you want to avoid CDC?

There are a couple of reasons. We see replication slots as a risk to the availability of the primary database. We have to take care of them when doing database upgrades. If we lose replication slots, we can't tolerate restarting the replication. Usually the data is updated rarely. If we add a new column, we may have to update all the data. This can be scheduled though.

without CDC, how do you plan to capture the relational database changes?

Triggers?

time based? then it’s kind of microbatch, you could fetch changelogs periodically

Triggers write to a table. Still transactional. PostgreSQL notifies application. Application fetches changes and pushes them. I prepared a working demo of this. Denormalization is also possible.

sure, you can give it a try and implement the kafka sink

I am not sure if I need Kafka. Scaling need is not much. Application can push to the target directly.

pinot needs kafka for ingestion