What are some ways to reduce the latency of stream...
# questions-and-troubleshooting
r
What are some ways to reduce the latency of streamload api. Currently I am using that API with batch of 15k records. Table details: • primary key table with 16 tablets (total 32 with replication) • Total rows in target table 500 million and columns 50 columns. • Table size if around 100GB. Current api latency is 1-1.5 mins which is too high. I am using partial update with row modes.
r
There's no easy way to determine the problem. You should see the profile and determine what caused the long wait https://docs.starrocks.io/docs/3.4/loading/loading_introduction/troubleshooting_loading/#diagnose-stream-load-performance Some suggestion, you could try to change it to column partial update if there's only small amount of columns are updated or increase the
load_mem_limit
on your load job (be careful when setting this limit, this could lead to OOM if not configured properly)