Hello guys,
Have anyone experience the _metadata s...
# troubleshootingy
Yang LI
02/21/2023, 9:18 AMHello guys,
Have anyone experience the _metadata size increase issue after flink migration? is there a solution or conf to fix it? π
β 1
Yang LI
02/21/2023, 10:51 AMsomehow it's kind of related to flink kafka source, if I change kafka operator uuid , it decrease the _metadata, somebody has a guideline about this kind of issue?
m
Martijn Visser
02/21/2023, 2:00 PMI would recommend checking if you have backpressure or potential idleness in your Kafka source https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/#idleness
π 1
y
Yang LI
02/22/2023, 2:49 PMHello @Martijn Visser, about this issue, we fixed the metadata size by following actions:
β’ change kafka connector operator uuid to discard old kafka connector state
β’ add conf state.storage.fs.memory-threshold: 1b to limit state into metadata.
So we understand we have much more inline state after migration, and we have much more files in our checkpoint.
Do you know any rocksdb conf to help us to clean or optimise checkpoint files to help us to reduce the file numbers in our checkpoint ? π
4 Views