use case is
it is reading raw data from pino realtime table And pusing it to offline table after doing some data clean up and mapping by connect to other database...just internal logic.
AS pinot does not support update column values..so we are doing so.
segment in offline table table we define as Segment as hourly.
one schedule job collecting the data each 15 minutes and uploading segment,,And pinot we are able to see incremental data.
when it becomes last hour..i.e 60th minutes our data will becomes 150 gb.
need on advice on this..As I am sure I am not able to load the segment in to pinot database.so what I can do.If i make 30 minute segment it becomes 75GB.
how i optimize the segment size..is there any limit on segment size?