Hello folks, Need your suggestions on flink clust...
# troubleshooting
s
Hello folks, Need your suggestions on flink cluster tuning, We are using flink to process data from IoT devices(140k), we are getting ~4Mbps of data in the kinesis stream(4 shards), a message from device has over 230 fields as a nested object, we are mapping into Paimon Row using built in serializers, and finally writing in to a single paimon table over s3, Using Flinksinkbuilder for writing rows, by looking at the job graph its evident that writer is becoming bottleneck and cpu utilisation hits 100%, along with data skew and builds back pressure also checkpoints take long time to process over period of time they start to fail. Currently use AWS managed flink services with 10 KPUs each kpu has 1vcpu that is 4gb of Memory and 30gb of storage Can somebody guide me in the right direction for tuning
p
Hello there! Have you tried increasing the parallelism of your writes?
s
Yes, but that dint help, global committer is blocking
p
global committer? Is the sink operation in a different chaining?