Hey everyone, I’ve recently started using the OSS ...
# troubleshooting
f
Hey everyone, I’ve recently started using the OSS Flink autoscaler and ran into a couple of major issues when handling larger jobs (1k+ operator parallelism and 3k+ topic partitions), specifically around fetching vertex-level metrics. Both issues appear to be related to performance issues in JM endpoints. More details in 🧵
1. /jobs/jobid/vertices/vertexid/subtasks/metrics – This endpoint is used by
ScalingMetricCollector::queryAggregatedMetricNames
to fetch subtask metric names for a given vertex. However, for large jobs (high parallelism, thousands of Kafka topics), this endpoint returns more than 40MB of data (just the metric names), which is unmanageable and causes timeouts. 2. /jobs/:jobid – This endpoint collects
numRecordsIn
,
numRecordsOut
, and
accumulatedBusyTime
. The problem here is that the JM needs to fetch this data from each TM for every subtask, then compute the sum across all subtasks. The current implementation in
MetricFetcherImpl::update
is synchronous and can take minutes, causing CPU and GC spikes in the JM—definitely not suitable for large jobs in prod.
Proposed Changes: 1. Optimize Subtask Metric Fetching: • I suggest adding a "list of regex" parameter to the
/jobs/:jobid/vertices/:vertexid/subtasks/metrics
endpoint. This way, we can match metrics based on patterns (e.g., metrics starting with "Source__" and ending with "bytes-in") rather than needing to retrieve all the metric names. This will reduce the payload size and significantly decrease the risk of timeouts, improving autoscaler efficiency. • Backward compatibility: If no regex is provided, the endpoint will continue to function as it did previously, ensuring no disruption for existing systems. 2. Fetch Metrics More Efficiently: • For the three metrics (
numRecordsIn
,
numRecordsOut
,
accumulatedBusyTime
), I propose we stop relying on
/jobs/:jobid
and instead fetch them in the same way we do for other metrics via
/jobs/:jobid/vertices/:vertexid/subtasks/metrics?get=accumulateBusyTimeMs,numRecordsIn,numRecordsOut
. These metrics are essentially the same across both endpoints, so I’m not sure why they’re handled differently. • Additionally, we could add a flag to
/jobs/:jobid
to prevent it from returning vertex-level metrics, allowing us to skip the slow fetching process entirely. What do you think? Any feedback would be greatly appreciated!
g
I think your suggestions make complete sense. I think if we want to proceed with this plan, we will need a FLIP as this touches public APIs and both flink core and the operator
👍 1
Also in general for these larger discussions / concrete proposals , the dev mailing list is a much better place for visibility
🆗 1