The documentation here (<docs.starrocks.io/docs/…/...
# questions-and-troubleshooting
m
The documentation here (docs.starrocks.io/docs/…/STREAM_LOAD#…) says that the merge_commit_* parameters only work for a single-db/single-table load. The documentation here (docs.starrocks.io/docs/loading/Stream_Load_transaction_interface) says it is possible to do multi-table stream-load. I had Codex write a little test trying to use the merge_commit parameters with multi-table stream load endpoints, and it failed at prepare. But, if the code skips prepare and goes directly to commit, it works with the merge_commit happening as hoped, (i.e. with a mix of values in the merge_condition column, the highest one always prevailed for a given PK, regardless of where the records with the highest merge_codition value was first, last or middle. NOTE1: We are running SR
4.1.0-4ee68c0
NOTE2: Here was the curl sequence:
1. begin:
curl --location-trusted -sS --max-time 30 \
-u "${STARROCKS_USER}:${STARROCKS_PASSWORD}" \
-H "label:codex_merge_headers_reordered_1792498469" \
-H "Expect:100-continue" \
-H "transaction_type:multi" \
-H "enable_merge_commit:true" \
-H "merge_commit_async:false" \
-H "merge_commit_interval_ms:5000" \
-H "merge_commit_parallel:2" \
-H "db:anchor" \
-H "table:A_reordered" \
-X POST \
"<http://chi-tdw-fe.hornblower.com:8030/api/transaction/begin>"
2. Load table A:
curl --location-trusted -sS --max-time 30 \
-u "${STARROCKS_USER}:${STARROCKS_PASSWORD}" \ -H "label:codex_merge_headers_reordered_1792498469" \ -H "Expect:100-continue" \ -H "transaction_type:multi" \ -H "enable_merge_commit:true" \ -H "merge_commit_async:false" \ -H "merge_commit_interval_ms:5000" \ -H "merge_commit_parallel:2" \ -H "db:anchor" \ -H "table:A_reordered" \ -H "format:csv" \ -H "column_separator:," \ -H "columns:sk,DM" \ -H "merge_condition:DM" \ --data-binary @/Users/matthew.thurmaier/Documents/Playground/stream-load-experiment-A.csv \ -X PUT \ "http://chi-tdw-fe.hornblower.com:8030/api/transaction/load" 3. Load table B: curl --location-trusted -sS --max-time 30 \ -u "${STARROCKS_USER}:${STARROCKS_PASSWORD}" \ -H "label:codex_merge_headers_reordered_1792498469" \ -H "Expect:100-continue" \ -H "transaction_type:multi" \ -H "enable_merge_commit:true" \ -H "merge_commit_async:false" \ -H "merge_commit_interval_ms:5000" \ -H "merge_commit_parallel:2" \ -H "db:anchor" \ -H "table:B_reordered" \ -H "format:csv" \ -H "column_separator:," \ -H "columns:sk,DM" \ -H "merge_condition:DM" \ --data-binary @/Users/matthew.thurmaier/Documents/Playground/stream-load-experiment-B.csv \ -X PUT \ "chi-tdw-fe.hornblower.com/api/transaction/load" 4. Commit (no prepare): curl --location-trusted -sS --max-time 30 \ -u "${STARROCKS_USER}:${STARROCKS_PASSWORD}" \ -H "label:codex_merge_headers_reordered_1792498469" \ -H "Expect:100-continue" \ -H "transaction_type:multi" \ -H "enable_merge_commit:true" \ -H "merge_commit_async:false" \ -H "merge_commit_interval_ms:5000" \ -H "merge_commit_parallel:2" \ -H "db:anchor" \ -X POST \ "chi-tdw-fe.hornblower.com/api/transaction/commit" QUESTIONs: 1. I believe this is not SUPPORTED behavior, though it is desireable. 2. Can someone who reads code better than me validate that this is infact what the code is doing, that the experiment results aren't just a fluke? 3. If the code does indeed do this, can it become supported and remain in the roadmap? PING.