Hi All. I’m using Flink to write data from Kafka t...
# troubleshooting
p
Hi All. I’m using Flink to write data from Kafka to iceberg tables in real time. The IcebergStreamWriter is working a lot slower than expected. I’m receiving around 50k records in 5 minutes while the records sent from stream writer are only 20 in the same time. I am appending the table and have put upsert as false. Can someone help me with the configs that I need to work around and check to improve the efficiency?
👀 1
t
Did you try running a profiler? We've had issues in the past with various things (such as slow metastore table refreshes)
f
If you can share your findings and how did younresolve it would be great.