https://pinot.apache.org/ logo
Join Slack
Powered by
Hey, I'm having an issue with hybrid tables. It lo...
# troubleshooting
d
Hey, I'm having an issue with hybrid tables. It looks like RealtimeToOfflineSegmentsTask was not moving all the records in the realtime segment, son for example I've got 4mill records in the realtime table but I end up with only 600 in the offline table after the realtime segments are deleted. Surely this is a configuration issue in my table. I was wondering whether this could be a problem between the table retention periods and the offline flow buffer configuration.
In fact the realtime segments in S3 are few MB each whereas the offline segments are nearly few KB
n
can you share both table configs?
the realtime to offline job will move data from realtime to offline, periodically, 1 window at a time. So if your realtime table has several time windows, it can take several runs before all data is moved over. If the realtime table’s retention kicks in before that, it will get deleted. The 2 processes aren’t aware of each other
d
Hi Neha, I've tried different configurations. I have 5 days of test data in my Kafka topic by the time the tables are created - I can see all the records in the realtime table before the realtime segments are deleted
Copy code
{
  "tableName": "some_table",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": 730,
    "timeColumnName": "posted_timestamp_utc",
    "timeType": "MILLISECONDS",
    "schemaName": "some_table",
    "replicasPerPartition": "1",
    "replication": 1
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "HOURLY"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
Copy code
{
    "tableName": "some_table",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "retentionTimeUnit": "HOURS",
      "retentionTimeValue": 12,
      "segmentPushType": "APPEND",
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "timeColumnName": "posted_timestamp_utc",
      "timeType": "MILLISECONDS",
      "schemaName": "some_table",
      "replicasPerPartition": "1"
    },
    "tenants": {},
    "tableIndexConfig": {
      "loadMode": "MMAP",
      "streamConfigs": {
        ...
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
        "group.id": "pinot-table-gorupid",
        ...
      }
    },
    "task": {
      "taskTypeConfigsMap": {
        "RealtimeToOfflineSegmentsTask": {
          "bucketTimePeriod": "1h",
          "bufferTimePeriod": "3h",
          "mergeType": "concat",
          "maxNumRecordsPerSegment": "1000000"
        }
      }
    },
    "metadata": {
      "customConfigs": {}
    }
  }
I've tried changing the retention period in the realtime time but that doesn't seem to change anything
I'm not sure whether there is some clash between the RealtimeToOfflineSegmentsTask settings and the batchIngestionConfig - I'm still not clear about how they play together if they do at all
I have a deployment with one node of each type at the moment. One thing I've just noticed is that when the server is redeployed the segments are moved to deep storage under DeletedSegments and the realtime table doesn't reprocess them - they won't be available to queries. I had to delete the table and recreate it to consume all the data again. Is this an expected behaviour? I was actually expecting the low level consumer to restart the ingestion from the earliest offset that still has not been moved to the offline table
n
Before the retention deletes data from the realtime table, what’s the min and max 
posted_timestamp_utc
? Can you do a 
select min(posted_timestamp_utc), max(posted_timestamp_utc) from some_table_REALTIME group by $segmentName order by $segmentName limit 1000
so we can observe the timestamps in each segment?
Once retention has deleted segments, they will not be used by Pinot for queries. That is expected behavior.
Looking at your configuration, the data from realtime will be moved 
1 hour
at a time, starting from the earliest timestamp available in the realtime table. The minion task runs hourly. So if you have x hours of data in the realtime table, it will take x hours to move it over to offline. Now if 
x < 12h
, the movement is not going to happen in time, and retention will delete them
🙌 1
So what’s the 
x
in your case?
I was actually expecting the low level consumer to restart the ingestion from the earliest offset that still has not been moved to the offline table
- the realtime consumption, the retention periodic task, and the realtimeToOffline periodic task are all independent of each other. They are not aware of the progress of the other. It is upto us, to configure retention such that the realtimeToOffline task gets enough time to move over the segments
d
Is that 
x > 12h
instead? I think it makes sense
👍 1
I'm still not sure about the ingestion. My question is actually what happens when a pinot realtime server goes down and back up? I think consumption should resume from where it's been left by reloading the consuming segments or consuming the segments again from the beginning?
Also, is there any the reason to move 1 window at the time? If there are 2 windows that could be moved when the task run it could make sense to move all/n in that same execution?
n
we maintain a watermark and slide forward. This is to simulate how an offline flow would have done batch ingestion. You can always trigger the job manually, during the bootstrap. There’s an API in swagger under /tasks. It will still be 1 at a time, but will not have to wait for hourly cadence
what’s the total data in the realtime table that you want to move?
when the server goes down, we begin consuming again from the last offset from the last completed segment. this video might help understand that

https://youtu.be/WoruCQgPhSA▾

d
Thank you Neha, I'm still new to Pinot and I'm trying to understand as much as possible. I've been looking at the data in ZK and I more or less understand what's in there
I've adjust the retention and offline flow settings and now is looking better
I'm in a test environment so the amount of data I want to move is small, few days which is few million records in total
Thanks for the video
n
cool! happy to help. since you went through this exercise, how could the Managed Offline Flows documentation have been different, so that it would’ve been clear from the start? would appreciate your feedback so we can make the doc better
d
Let me read through it again and I'll get back to you if can think of some suggestions 👍
Hi Neha, here are some suggestions for the documentation: 1) I would add a reminder that the window is relative to the timestamp configures in 
segmentsConfig
2) One issue I faced was the retention task getting stuck in running (I don't know why but it happened a couple of times) so no other tasks would be scheduled. This caused retention getting rid of realtime segments and data not making it to the offline table. I used the task API to delete the tasks and then everything went back to normal. I only noticed this when I was sinking every hour - not sure if it is worth adding a troubleshooting section with some advice on where to look and what to do when something like this happens 3) The docs says 
controller.task.frequencyInSeconds
defaults to -1 which is disabled. I had to look at the source code to understand how this controls the task scheduler. Not sure if this should be mentioned in the offline flows doc but a description in the controller configuration would be great. 4) I had to configure deep storage in the minion as well. These settings don't seem to follow the controller/server conventions so it may be work in progress? They are not described in the deep storage configuration doc either
n
This is great feedback, thank you so much. Will fix these things this week
d
Thank you Neha!
2 Views
Open in Slack
PreviousNext
Powered by