We have a FlinkSQL join operator that's running really slow (about 1k every few minutes). It's 100% busy. We're on pretty large hardware and the parallelism is set to 8. The join is simple, just on two columns. I'm not exactly sure how to find the root cause. Here is the flame graph. Seems like most time is spent in rocksdb. Any recommendations on what this can be and/or where to look?

Can you please send RocksDB logs (preferably with statistics enabled)? Are you doing a lot of deletes?

If you're not careful, sometimes joins can be pathological in terms of their resource requirements. What does the join look like?