This message was deleted.
# generals
Slackbot
05/04/2023, 3:18 PMThis message was deleted.
s
Sergio Ferragut
05/04/2023, 3:38 PMNot sure why the historicals are failing. Looking at their logs should shed some light. The speed with which historicals will load segments depends on a few factors. Here's a recent thread on this. Also, it makes sense to use a stateful set for the historicals such that their local storage of cached segments files is persistent across restarts.
r
Remya Savithry
05/04/2023, 3:58 PMHello Sergio,
Thanks for the reply.
But I am unable to open the link you shared.
d
David McHealy
05/05/2023, 1:06 PMDo the logs indicate the historicals are loading from s3 on each start?
David McHealy
05/05/2023, 1:09 PMI ask that because 6 hours for that much data with even just one historical seems like a long time if the data already exists in a k8s volume on disk after a prior restart.
g
Gian Merlino
05/10/2023, 5:51 PM6 hours does seem like a super-excessive amount of time to load data from an already-existing PV
Gian Merlino
05/10/2023, 5:52 PMI wonder what it's logging during this time?
You probably see a bunch of lines like
Loading segment cache file [%d/%d]Gian Merlino
05/10/2023, 5:52 PMNormally a historical would have up to tens of thousands of those (possibly low-hundreds-of-thousands for very dense historicals; rarely more)
Gian Merlino
05/10/2023, 5:53 PMand typically they go by pretty quickly
Gian Merlino
05/10/2023, 5:53 PMwhile this is happening do you see I/O maxing out on your PVs?
r
Remya Savithry
05/24/2023, 3:52 AMHello @Gian Merlino,
As you mentioned above we can see the Loading segment cache file [%d/%d] and Loading segment for data source in the historical logs during this time.
At some point historical pods restart and it will start loading the segment cache files from the beginning.
This cycle will repeat and will never ends.
Finally we ended up in clearing all data sources and started loading them from Kafka again.
Remya Savithry
05/24/2023, 4:01 AMHello @David Smith,
We are using the Azure storage account as the deep storage.
We didn't get anything from the logs to confirm it is loading the segments from the deep storage on each start.
But we can see two series of logs in the historical pods. The first is 'Loading segment Cache file' and the second is loading segment
d
David McHealy
05/24/2023, 2:11 PMIgnore literally everything I just said. It just looks like you have a ton of segments, so it takes a long time.
s
Sergio Ferragut
05/24/2023, 6:38 PMA few of thoughts:
• it would be good to understand why the historicals are failing. Is this kubernetes? Is kubernetes killing the pods because they are using too many resources? If so changing the resources available to each historical pod should help, or changing their config to use less resources.
• Perhaps you've already done this, but if not segment size optimization is always good to pay attention to.
• Historicals should not need to reload all of their segments after a quick restart. If you are using kubernetes, deploy them as a stateful set. Relevant info toward the last paragraph in this section of the docs. Essentially the historical pod that restarts looks at its local segment cache to announce the segments it already has. If you use statefulset for historicals, their local storage should still have the segments they need after the restart.
d
David McHealy
05/24/2023, 7:03 PMI don't think it is loading from s3, based on the screenshot. But it is loading about five segments per second. That is 4.5 hours to pull its segments from its cache. My best guess is whatever volumes you have mounted are very, very slow.
David McHealy
05/24/2023, 7:04 PMOr the segments are very, very large.
2 Views