At which point does INSERT INTO VALUES queries no longer make sense over other methods of loading? I'm working with batches of 5000~50000 rows at a time here, about 1 second apart.
The main issue here for me is that these batches are across multiple tables so I'm using the BEGIN TRANSACTION ->INSERT INTO ->COMMIT stages of loading, but for row batches up only ~5~10MB in string length in the insert query, the FE memory spikes several gigabytes and starts giving out OOM kill signals. Is this sort of load better suited for Stream Loads/Routine Loads (May not be an option)?
Would I run into similar OOM errors in the FE if I used Stream Loading for instance because it also just tries to do a mass insert of rows under the hood?