Apache Pinot

Hello,
I’m using RealtimeToOfflineSegmentTask to move segments from REALTIME tables to OFFLINE ones, it works pretty well on some of my tables, but 2 of them are ignored. I can’t find anything in the logs.
If I check the value of
```&lt;cluster name&gt;/PROPERTYSTORE/MINION_TASK_METADATA/RealtimeToOfflineSegmentsTask/&lt;table name&gt;```
in Zookeeper, I see that the document hasn’t been updated since August 2nd. And I can’t found any task related to this table in
```&lt;cluster name&gt;/CONFIGS/RESOURCE/TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1632701552537```
How is this possible? How can I reset this task? There is somewhere I can look for infos to debug this and prevent it from happening again?
I’m using Pinot 0.7.1 (I plan to upgrade soon but I can’t right now)

can you check the min/max of the time column in each segment

```start: 1632454589759
end: 1632541000579

start: 1632541000917
end: 1632627410395

start: 1632627410714
end: 1632713820512```
for completed segment in the realtime table

and the task config the in table is:
```"task": {
      "taskTypeConfigsMap": {
        "RealtimeToOfflineSegmentsTask": {
          "bucketTimePeriod": "2d",
          "bufferTimePeriod": "1d",
          "collectorType": "concat",
          "maxNumRecordsPerSegment": "390000"
        }
      }
    },```

Check for logs in the controller. There should be a log about trying to schedule the task

Right, thank you. Found it
```2021/09/28 08:13:29.014 INFO [RealtimeToOfflineSegmentsTaskGenerator] [pool-9-thread-1] Start generating task configs for table: datasource_60db269da8ed00030004fa86_REALTIME for task: RealtimeToOfflineSegmentsTask
2021/09/28 08:13:29.019 INFO [RealtimeToOfflineSegmentsTaskGenerator] [pool-9-thread-1] Found no eligible segments for task: RealtimeToOfflineSegmentsTask with window [1627689600000 - 1627862400000). Skipping task generation```
Since the watermark is stuck on July 31, Pinot is always trying to find segments between July 31 and August 2nd, but since there is no segment (we delete them from the REALTIME table every 4d) the task is “stuck” and never try to move new available segments
Is this the expected behavior? That’s quite annoying because it means that the task could be stuck if there is no available segment in an old window

this is a known issue. <https://github.com/apache/pinot/issues/6988> Typically you wouldn’t find such gaps in the data in a production setup, and so this was by design made to wait for events in the window it’s operating on. Until we fix this, you can manually edit the watermark in the minion metadata znode to the next window with available data

Thank you. We’re using Pinot in production and we have this issue since we create a table for each of our customer and some of them don’t write data often which can lead to this issue.
I will try updating the watermark in the minion metadata but it isn’t a viable solution for us :confused:

I see, in that case will try to prioritize the fix. Contributions welcome!

Thank you, for you what’s the best workaround right now? Pushing a row every day to ensure segment creation or a cron that update the zookeeper data?

either is fine. The former is less risky/error prone. In the second case, you dont want to accidentaly end up skipping the existing rows