Keyur Makwana
06/21/2023, 1:07 PMKeyur Makwana
06/21/2023, 1:07 PMMikhail Spirin
06/21/2023, 1:30 PMJoe Karayusuf
06/21/2023, 3:36 PMKeyur Makwana
06/22/2023, 6:46 AMKeyur Makwana
06/22/2023, 6:48 AMJoe Karayusuf
06/22/2023, 12:52 PMJoe Karayusuf
06/22/2023, 12:53 PMJoe Karayusuf
06/22/2023, 12:53 PMJoe Karayusuf
06/22/2023, 1:14 PMKeyur Makwana
06/22/2023, 1:49 PMKeyur Makwana
06/26/2023, 4:16 AMJoe Karayusuf
06/26/2023, 1:56 PM.rebalance between the co-process and sink function. Rebalances aren't cheap. They require flink to serialize the data, send it over the network, and then deserialize the data. You're rebalancing between the co-process function and the sink. Since the data is already rebalanced going into the co-process function, the second rebalance probably isn't doing much.
2. Once the rebalance is removed, try enabling object reuse. env.getConfig.enableObjectReuse . Assuming you're using immutable objects, this should be a nice performance win. It'll ensure that flink doesn't serialize and deserialize the objects between the co-processor and sink when they are in the same task group.Joe Karayusuf
06/26/2023, 2:00 PMJoe Karayusuf
06/26/2023, 2:02 PMcirce to jsoniter-scale in our flink app and saw a huge performance improvement. I imagine switching from jackson or gson to jsoniter would show similar benefits, but can't confirm that.Joe Karayusuf
06/26/2023, 2:04 PMKeyur Makwana
06/26/2023, 2:06 PMKeyur Makwana
06/26/2023, 2:07 PMJoe Karayusuf
06/26/2023, 2:08 PMAsyncSinkBase. If not, take a look at implementing one.
2. Are you batching data to the sink? If not, I'd recommend batching the data to reduce the number of calls downstream.Keyur Makwana
06/26/2023, 2:08 PMJoe Karayusuf
06/26/2023, 2:11 PMKafka, Kinesis , something else?Keyur Makwana
06/26/2023, 2:12 PMKeyur Makwana
06/27/2023, 1:15 PMJoe Karayusuf
06/27/2023, 2:26 PMKeyur Makwana
06/27/2023, 2:47 PMKeyur Makwana
06/28/2023, 3:45 PMKeyur Makwana
06/28/2023, 3:46 PMKeyur Makwana
07/02/2023, 5:51 PMJoe Karayusuf
07/05/2023, 2:35 PMKeyur Makwana
07/06/2023, 5:02 AM