https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, i've created realtime to offline flow. i...
# general
v
Hi team, i've created realtime to offline flow. i've pushed 2k data points to realtime table. 1800 data points has been moved to offline table but same data i can see in realtime tables as well. how to remove those data from realtime table?
n
They will be removed when the retention of the table is reached, as per what you've set in retention time value
https://docs.pinot.apache.org/basics/components/broker querying will take care of not reading double data
v
Thanks @Neha Pawar
https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime in this article there is one there is one parameter
Copy code
retentionHours
can you please help me where should i add that in realtime table config?
n
v
Thanks @Neha Pawar, i've tried retentionTimeUnit and retentionTimeValue with "HOURS" and "1" value but the problem is, it is deleting all the realtime data including those which didn't move to offline table. so we may lose those data.
n
you shouldnt set it to 1 HOURS. typically, set it to something higher than the realtime to offline bucket + buffer
e.g. if buffer is 2h, bucket is 1d, minion cron is set to hourly. then set retention to 2d
v
yeah, we will set in month but we are just testing as of now
one more issue i found out is i am able to see completed segment in realtime but its not pushing to offline till i push new data to realtime
@Seunghyun can you please add any comment on this?
s
what's your config for the minion task?
v
Copy code
"task": {
        "taskTypeConfigsMap": {
            "RealtimeToOfflineSegmentsTask": {
                "bufferTimePeriod": "3m",
                "bucketTimePeriod": "5m",
                "schedule": "0 */1 * * * ?",
                "mergeType": "dedup",
                "maxNumRecordsPerSegment": "10"
            }
        }
    },
s
The task should start to schedule completed segments after 
bufferTimePeriod
is passed
Copy code
one more issue i found out is i am able to see completed segment in realtime but its not pushing to offline till i push new data to realtime
v
yeah, bufferTimePeriod is passed
s
also, bufferTimePeriod/bucketTimePeriod is too small for the production (i assume that you're just testing)
Copy code
"schedule": "0 */1 * * * ?",
how often this should schedule the job? every minute?
v
yeah because of testing only
yeah every minute
s
in that case, we probably need to check the logs to see what's happening
if you can dump the controller log, it's probably going to be helpful
v
okay adding here, 1 min
Copy code
Execute CronJob: table - tab5_REALTIME, task - RealtimeToOfflineSegmentsTask at 2022-11-17T08:22:00.001+0000
Start generating task configs for table: tab5_REALTIME for task: RealtimeToOfflineSegmentsTask
Window data overflows into CONSUMING segments for partition of segment: tab5__0__15__20221117T0810Z. Skipping task generation: RealtimeToOfflineSegmentsTask
Finished CronJob: table - tab5_REALTIME, task - RealtimeToOfflineSegmentsTask, next runtime is 2022-11-17T08:23:00.000+0000
as per this segments are not completed, but i can able to see those are completed in zookeper browser
image.png
s
can you provide the IDEALSTATE
state for that segment?
v
Copy code
"tab5__0__14__20221117T0810Z": {
      "Server_pinot-server-0.pinot-server-headless.default.svc.cluster.local_8098": "ONLINE"
    },
Hi @Seunghyun, can you please help me with this?
s
Copy code
// Check overlap with window
if (windowStartMs <= segmentEndTimeMs && segmentStartTimeMs < windowEndMs) {
  // If last completed segment is being used, make sure that segment crosses over end of window.
  // In the absence of this check, CONSUMING segments could contain some portion of the window. That data
  // would be skipped forever.
  if (lastLLCSegmentPerPartition.contains(segmentName) && segmentEndTimeMs < windowEndMs) {
    <http://LOGGER.info|LOGGER.info>("Window data overflows into CONSUMING segments for partition of segment: {}. Skipping task "
        + "generation: {}", segmentName, taskType);
    skipGenerate = true;
    break;
  }
  segmentNames.add(segmentName);
  downloadURLs.add(segmentZKMetadata.getDownloadUrl());
}
I think that we are hitting this check but in production, the realtime-to-offline will continuously move data to offline correctly because there will be data flow in constantly. Also, we always query both offline/realtime tables so query result correctness will always be the same. So, I would say that it’s behaving as expected but we are not promoting the last bucket to offline because 1. last completed segment’s end time is less than windowEndMs 2. No more data after this so that there’s no event that would trigger the window bump
v
but total segments are 50 and 10 only consuming it. but 40 are not moving untill we push new data
Open in Slack
PreviousNext
Powered by