Francesco
02/03/2025, 9:32 PMFrancesco
02/03/2025, 9:33 PMScalingMetricCollector::queryAggregatedMetricNames to fetch subtask metric names for a given vertex. However, for large jobs (high parallelism, thousands of Kafka topics), this endpoint returns more than 40MB of data (just the metric names), which is unmanageable and causes timeouts.
2. /jobs/:jobid – This endpoint collects numRecordsIn, numRecordsOut, and accumulatedBusyTime. The problem here is that the JM needs to fetch this data from each TM for every subtask, then compute the sum across all subtasks. The current implementation in MetricFetcherImpl::update is synchronous and can take minutes, causing CPU and GC spikes in the JM—definitely not suitable for large jobs in prod.Francesco
02/03/2025, 9:34 PM/jobs/:jobid/vertices/:vertexid/subtasks/metrics endpoint. This way, we can match metrics based on patterns (e.g., metrics starting with "Source__" and ending with "bytes-in") rather than needing to retrieve all the metric names. This will reduce the payload size and significantly decrease the risk of timeouts, improving autoscaler efficiency.
• Backward compatibility: If no regex is provided, the endpoint will continue to function as it did previously, ensuring no disruption for existing systems.
2. Fetch Metrics More Efficiently:
• For the three metrics (numRecordsIn, numRecordsOut, accumulatedBusyTime), I propose we stop relying on /jobs/:jobid and instead fetch them in the same way we do for other metrics via /jobs/:jobid/vertices/:vertexid/subtasks/metrics?get=accumulateBusyTimeMs,numRecordsIn,numRecordsOut. These metrics are essentially the same across both endpoints, so I’m not sure why they’re handled differently.
• Additionally, we could add a flag to /jobs/:jobid to prevent it from returning vertex-level metrics, allowing us to skip the slow fetching process entirely.
What do you think? Any feedback would be greatly appreciated!Gyula Fóra
02/04/2025, 6:31 AMGyula Fóra
02/04/2025, 6:32 AM