Hey all, wondering if you could provide me some gu...
# troubleshootings
Sandip Nayak
08/19/2023, 11:53 PMHey all, wondering if you could provide me some guidance towards improving Table API JDBC sink performance, I am running a job which joins a Kafka topic with data stream from two different s3 sources, it seems like the
SinkMaterializersink.buffer-flush.max-rowssink.buffer-flush.intervalSinkMaterializerSandip Nayak
08/19/2023, 11:53 PMFlink version
1.16.2Sandip Nayak
08/19/2023, 11:55 PMI am no way an expert analyzing Flame graph, but this is what I see
Sandip Nayak
08/19/2023, 11:55 PMSandip Nayak
08/19/2023, 11:56 PMNot sure how to interpret that
Sandip Nayak
08/19/2023, 11:56 PMAny help would be appreciated!
s
sap1ens
08/21/2023, 12:37 AMSinkMaterializer used a lot of state, so you need to look into optimizing that. Do you use RocksDB for your state backend?
gratitude thank you 1
s
Sandip Nayak
08/21/2023, 1:24 AMthanks, yes we are using
EmbeddedRocksDBStateBackendSandip Nayak
08/21/2023, 1:27 AMbeside this https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/large_state_tuning/, wondering if you have any other places I can look into for large state optimizations
s
sap1ens
08/21/2023, 4:15 AMThe disks used for RocksDB storage can make a huge difference. You want to obtain the fastest SSD you can get, ideally instance-level (e.g. EBS on AWS can be too slow). Combine that with
state.backend.rocksdb.predefined-options: FLASH_SSD_OPTIMIZEDβ
1
π 1
j
jbfletch
08/28/2023, 2:05 PMI wonder if SpeedDb would be any faster, we have been benchmarking with kstreams for heavy state and it has shown pretty big improvements.
π 3
s
Sandip Nayak
08/28/2023, 6:26 PMHow easy is it to integrate SpeedDb with Flink?
Sandip Nayak
08/28/2023, 6:29 PMI want to give it a try for sure, but does not look like there is a great deal of integration information available
s
sap1ens
08/28/2023, 6:47 PMIt should be a drop-in replacement
β
1
j
jbfletch
08/28/2023, 6:52 PMitβs totes a drop in replacement https://github.com/speedb-io/speedb/releases π be interested to hear back if you swap
π 2
2 Views