Slackbot
10/11/2023, 5:59 AMBibek Sahoo
10/11/2023, 7:33 AM1) Fetch the list of partitions from Kafka and determine the starting offset for each partition (either based on the last processed offset if continuing, or starting from the beginning or ending of the stream if this is a new topic).
2) Discover any running indexing tasks that are writing to the supervisor's datasource and adopt them if they match the supervisor's configuration, else signal them to stop.
3) Send a status request to each supervised task to update our view of the state of the tasks under our supervision.
4) Handle tasks that have exceeded taskDuration and should transition from the reading to publishing state.
5) Handle tasks that have finished publishing and signal redundant replica tasks to stop.
6) Handle tasks that have failed and clean up the supervisor's internal state.
7) Compare the list of healthy tasks to the requested taskCount and replicas configurations and create additional tasks if required in case supervisor is not idle.Enoch Park
10/11/2023, 7:35 AM