Hi Team , there is a workaround to fix data gap i...
# troubleshooting
d
Hi Team , there is a workaround to fix data gap issue temporarily - https://github.com/apache/pinot/issues/6988 . Update watermark manually from where the data is available in realtime segment.It will move data from realtime table to offline table when the task is scheduled. Is it a feasible way to do this approach
m
Thanks @Deepak Mishra. You may want to add the details on how to update the watermark manually in the GH issue, for community members who might not know how to do so.
j
I'd prefer handling it in the task scheduler. It will be hard to manage if we expect user to find the gap and manually bump up the watermark
Will take a closer look at the code and see how we can handle it in the task scheduler. Shouldn't be hard since the data gap is handled within the merge-rollup task