How can I optimize the throughput of compaction fo...
# questions-and-troubleshooting
b
How can I optimize the throughput of compaction for a single table? Here is everything I've tried tuning. Setup is 12 CN nodes, with high amounts of JSON data (flat_json_enabled). Compaction starts falling behind at around ~400k rows/s, and I see that a majority of the time spent is on writing to S3, which blocks further compaction on the partition from beginning.
Copy code
min_cumulative_compaction_num_singleton_deltas = 10
compact_threads = 30
compact_thread_pool_queue_size = 100
max_cumulative_compaction_num_singleton_deltas= 100
max_base_compaction_num_singleton_deltas = 100
base_compaction_check_interval_seconds = 10
cumulative_compaction_num_threads_per_disk = 30
base_compaction_num_threads_per_disk = 30
cumulative_compaction_check_interval_seconds = 2
lake_compaction_stream_buffer_size_bytes = 10194304
pindex_major_compaction_limit_per_disk = 8
lake_pk_compaction_max_input_rowsets = 5000
update_compaction_per_tablet_min_interval_seconds = 5
alter_tablet_worker_count = 30
max_download_speed_kbps = 200000