StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. StarRocks received InfoWorld’s 2023 BOSSIE Award for best open source software.

StarRocks

Hello! Out of curiosity, what is the technical reason for why json streaming may lead to huge memory consumption above 100MB?

&gt; By default, the size of the JSON body in an HTTP request cannot exceed 100 MB. If the JSON body exceeds 100 MB in size, an error "The size of this batch exceed the max size [104857600] of json type data data [8617627793]. Set ignore_json_size to skip check, although it may lead huge memory consuming." is reported.
from <https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD/>

I'm testing out the best way to load tens of terabytes of JSON data into a starrocks shared data cluster and 100MB at a time is going to take a while. My CNs have a max of 16GB of memory at the moment and I got some OOM issues when loading chunks of 1GB, so I'm curious as to how this json parsing works, and whether there's a better solution.

An alternative is to convert that json to columnar parquet data and then read that with FILES(...) in starrocks, but if there's a direct JSON conversion into StarRocks tables, that'd be amazing. Thanks! :blush: