Hi, I am trying to do batch ingestion via Spark to a OFFLINE table. It often crashed the broker when the data is large enough. From what I can tell, larger memory allocated to broker can help. But Is there any documents about how to calculate best settings about what should be right memory allocation, disk size, etc for different type of nodes ? For example for 1T data ?

Batch ingestion should not crash broker due to data size. Do you have too many segments being pushed?