@Rocky using the merge commit with stream load works well but I do lose fidelity on tracking, I noticed PROFILE_ID in

information_schema.loads

, what is that? I'm asking because I'm seeing HTTP timeouts on loads that are successful and I don't want to retry

The

PROFILE_ID

column in

information_schema.loads

is the unique identifier for the runtime profile of a load job. What is

PROFILE_ID

? It is used for deep performance diagnostics. If you have a specific

PROFILE_ID

, you can run the following command in your MySQL client to see exactly where the load spent its time (e.g., scanning, RPC, or writing to disk):

sql
ANALYZE PROFILE FROM 'your_profile_id';

_Note: You must enable load profiling for the table or session first (e.g.,

SET enable_load_profile = true;

) for this to be populated._ ────────── Solving Tracking and HTTP Timeouts If you are seeing HTTP timeouts but the load might be successful, you should not automatically retry. Instead, use the Label you assigned to the Stream Load to verify its state. 1. Check Status via Label: Before retrying, query

information_schema.loads

using the label you used for the request:

sql
    SELECT STATE, REJECTED_RECORD_PATH, ERROR_MSG
    FROM information_schema.loads
    WHERE LABEL = 'your_label_here';

* If

STATE

is

FINISHED

or

COMMITTED

, the load succeeded. * If

STATE

is

PREPARED

(in a 2PC context), the data is safely in StarRocks but the commit hasn't finalized yet; you should send a

commit

RPC rather than a new load. * If no record exists or the state is

CANCELLED

, it is safe to retry. 2. Why use

information_schema.loads

? As you noted, it provides better "fidelity" because it survives the HTTP timeout. Even if your client loses the connection, the FE (Frontend) continues the transaction. This view allows you to see: * `SINK_ROWS`: How many rows actually made it in. * `ERROR_MSG`: Why it failed if it did. * `TRACKING_SQL`: A SQL statement you can run to see exactly which rows were filtered out due to quality issues. Summary for your workflow: To handle timeouts without losing tracking: 1. Always use a unique Label for every Stream Load. 2. If you get an HTTP Timeout, catch the exception and query

information_schema.loads WHERE LABEL = ...

. 3. Only retry if the

STATE

is clearly

CANCELLED

or if the label is not found (and you've waited long enough for it to register). Reference: information_schema.loads References • sql-reference/information_schema/loads.md • sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md

@Rocky that doesn't work with merge commit as it creates its own label

The

PROFILE_ID

in

information_schema.loads

is the unique identifier for the Runtime Profile of the load job. It is used exclusively for performance tuning and diagnostics via the

ANALYZE PROFILE FROM '<profile_id>';

command. It does not serve as a tracking ID for individual data chunks within a merged transaction. Why you "lose fidelity" with Merge Commit When you enable

merge_commit

, StarRocks intentionally ignores the

Label

you provide in the HTTP header. It does this so it can bundle multiple independent requests (from different clients or threads) into a single large transaction with a system-generated label. Because your specific request is now part of a shared transaction: 1. Deduplication is lost: Since your label is ignored, StarRocks cannot use it to prevent duplicates if you retry the same request. 2. Tracking is difficult: If your HTTP client timeouts, you don't know the system-generated label or

txn_id

that your data was assigned to. How to track success after a timeout If you encounter an HTTP timeout while using

merge_commit

, you can't query by your own label. Instead, you can use

information_schema.loads

to look for the merged transaction that likely included your data: 1. Filter by Table and Time:

sql
    SELECT LABEL, STATE, CREATE_TIME, RUNTIME_DETAILS
    FROM information_schema.loads
    WHERE TABLE_NAME = 'your_table'
      AND TYPE = 'STREAM'
      AND CREATE_TIME >= 'time_just_before_your_request'
    ORDER BY CREATE_TIME DESC;

2. Use

RUNTIME_DETAILS

: This column contains a JSON string with the

client_ip

. If you are loading from a known set of clients, you can use this to see if a load from your IP was processed during the timeout window. 3. Check the

STATE

: * If a load covering your timestamp is

FINISHED

, your data was likely committed. * Warning: Because

merge_commit

only guarantees at-least-once semantics, if you cannot definitively confirm success and you retry, you may end up with duplicate data. Recommendation * If you need high-fidelity tracking (Exactly-Once): Do not use

merge_commit

. Instead, batch your data on the client side (e.g., to 100MB+ chunks) and use standard Stream Load with your own unique

Label

. This allows you to use

SHOW LOAD WHERE LABEL = '...'

to verify status even after a timeout. * If you need high-concurrency small loads: Stick with

merge_commit

, but ensure your application can handle potential duplicates (e.g., using a Primary Key table) or use a "wait and check" logic by querying the latest successful load time in

information_schema.loads

. References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/StreamLoad.md • sql-reference/information_schema/loads.md References • loading/loading_introduction/troubleshooting_loading.md • sql-reference/information_schema/loads.md

@Rocky I'm using nifi for this, if i wanted to get back to tracking without merge commits, should I limit nifi to sending one request at a time and spacing it out?

To get back to reliable tracking and Exactly-Once semantics without using

merge_commit

, you should transition to a batching strategy in NiFi. Simply "limiting NiFi to one request and spacing it out" is only half the battle; the key to tracking is batch size and labels. 1. The Recommended Strategy: NiFi

MergeContent

Instead of sending every record or small FlowFile as an individual Stream Load, use the

MergeContent

processor in NiFi before your StarRocks loading processor. * Batching: Set

MergeContent

to group FlowFiles by size (e.g., 50MB–100MB) or time (e.g., every 5–10 seconds). This reduces the number of transactions and prevents "too many versions" errors in StarRocks. * Unique Label: For the merged FlowFile, generate a unique

label

(e.g., using

${UUID()}

or

${filename}

) and pass it in the HTTP Header of your Stream Load (

-H "label: ${label_name}"

). 2. Why this solves your tracking/timeout issue When you use your own

label

with standard Stream Load (not merge commit): 1. Deduplication: If an HTTP timeout occurs, StarRocks might still be processing the data. If you retry with the same label, StarRocks will reject the retry with a

Label Already Exists

error rather than inserting the data twice. 2. Verification: You can query the status of that specific batch immediately after a timeout:

sql
    SELECT STATE, ERROR_MSG
    FROM information_schema.loads
    WHERE LABEL = 'your_nifi_generated_label';

If it shows

FINISHED

, you can safely move on in NiFi without retrying. 3. Should you limit NiFi to "One at a time"? * Concurrency: You don't necessarily need to limit NiFi to a single thread (

Max Concurrent Tasks = 1

) if your batches are large (100MB+). StarRocks can handle hundreds of concurrent transactions. * Spacing: StarRocks recommends at least 5 seconds between commits per table to maintain healthy compaction. `MergeContent`'s "Max Bin Age" is the perfect place to enforce this. * When to use Serial (1 thread): Only limit to one request at a time if you are loading into a Primary Key table and require strict ordering (e.g., CDC data where an update must happen after an insert). Summary of NiFi Workflow 1.

MergeContent

: Create batches of 50–100MB or wait 10 seconds. 2.

UpdateAttribute

: Create a

starrocks_label

attribute using

${UUID()}

. 3.

InvokeHTTP

(or similar): Execute Stream Load with

-H "label: ${starrocks_label}"

. 4. Error Handling: If a timeout occurs, route the FlowFile to a processor that queries

information_schema.loads

by that label before deciding to retry or fail. References • sql-reference/information_schema/loads.md • loading/StreamLoad.md • loading/loading_introduction/loading_concepts.md References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/Flink-connector-starrocks.md • faq/loading/Loading_faq.md • loading/Stream_Load_transaction_interface.md • administration/management/monitoring/metrics.md • loading/loading_introduction/loading_concepts.md

@Rocky I wasn't thinking about changing this

Max Concurrent Tasks = 1

but i have a cluster of 6 nodes, can i send 6 request to the same table at the same time?

Yes, you can absolutely send 6 concurrent requests to the same table. StarRocks is designed to handle multiple concurrent write transactions. However, there is a technical distinction between concurrency (how many loads are running at once) and frequency (how often you commit). 1. Concurrency (Safe) * The Limit: By default, StarRocks allows 100 concurrent transactions per database (controlled by the FE config

max_running_txn_num_per_db

). * Your Case: Sending 6 concurrent requests from your NiFi nodes is well within this limit. StarRocks will process them in parallel across your BE nodes. 2. Frequency (The Real Bottleneck) The reason you might have been using

merge_commit

originally is to solve the "Too Many Versions" problem. * Every successful Stream Load creates a new "version" of the data for the affected tablets. * If your 6 NiFi nodes each send a small request every 1 second, you are generating 6 versions per second. * StarRocks' background compaction needs time to merge these versions. If you create them faster than they can be merged, you will eventually hit the error:

too many tablet versions, build rowset failed

. Recommendations for your 6-node NiFi Cluster: To keep your high-fidelity tracking without

merge_commit

and without crashing the cluster: 1. Batch on the NiFi side: Ensure each of your 6 NiFi nodes is using a

MergeContent

processor. Instead of sending data as soon as it arrives, have each node wait until it has ~50MB–100MB of data OR 5–10 seconds have passed. 2. Staggered Commits: With 6 nodes each committing every 10 seconds, the table will receive roughly 1 commit every 1.5 seconds. This is a very healthy "cadence" that StarRocks can easily compact. 3. Unique Labels: As discussed before, ensure each NiFi node generates a unique label for its batch. This is your "source of truth" for tracking after a timeout. 4. Monitor Versions: You can check if your 6-node concurrency is causing issues by running:

sql
    -- Check if any tablets have too many versions (approaching 1000)
    SHOW TABLET FROM your_table_name;

Summary: 6 concurrent requests are perfectly fine and standard practice. Just ensure each request is a meaningful "batch" of data rather than a few rows, to keep the background version count healthy. References • faq/loading/Loading_faq.md • administration/management/FE_configuration.md • loading/loading_introduction/loading_concepts.md References • faq/loading/Loading_faq.md • faq/loading/Routine_load_faq.md • faq/shared_data_faq.md • faq/Others.md • deployment/environment_configurations.md • administration/management/FE_configuration.md

@Rocky this is the error I keep getting

org.apache.nifi.processor.exception.ProcessException: IOException thrown from InvokeHTTP[id=3485469f-2975-324b-80f0-67facf86fc1c]: java.net.SocketException: Connection reset by peer

I've updated the feproxy LB timeout to 20 minutes so that shouldn't be it even though the nginx timeout is 600 that still seems like plenty of time. Any other ideas?

The

Connection reset by peer

error in NiFi's

InvokeHTTP

generally means the StarRocks side (or your proxy) closed the TCP connection while NiFi was still trying to send data or wait for a response. Since you've already checked the 20-minute LB timeout, here are the most likely remaining culprits: 1. The "Early Rejection" (Most Common) If StarRocks encounters an error before it finished reading the entire data stream from NiFi, it will close the connection. NiFi, still trying to push the rest of the file, sees this as a "Connection Reset." * Check FE/BE Logs: Look at

fe/log/fe.warn.log

or

be/log/be.INFO

on the node you are hitting. Common causes for early rejection: * Label already exists: If NiFi retries a label that is still in "PREPARE" or "COMMITTED" status. * Table/Database not found: A typo in the URL. * Authentication failure: Invalid credentials. * JSON size limit: By default, StarRocks limits JSON bodies to 100MB. If your NiFi batch is larger, you must add the header

ignore_json_size: true

. 2. Lack of

Expect: 100-continue

StarRocks documentation explicitly recommends using the

Expect: 100-continue

header for Stream Load. * Why: It allows NiFi to ask the server "Are you ready to receive this large file?" before actually sending it. If StarRocks has an error (like the ones above), it can tell NiFi before the data transfer starts. * Fix: Add an HTTP Header in `InvokeHTTP`:

Expect: 100-continue

. 3. Nginx

proxy_request_buffering

If you are using Nginx as a proxy/LB, it might be trying to buffer the entire NiFi request to disk before sending it to StarRocks. * The Issue: If the transfer from NiFi to Nginx takes longer than Nginx's internal

client_body_timeout

, Nginx resets the connection. * Fix: Disable buffering in your Nginx config for the StarRocks location:

nginx
    proxy_request_buffering off;
    proxy_http_version 1.1;
    client_max_body_size 0; # Ensure no limit on load size

4. StarRocks Internal Timeouts Even if your LB is set to 20 minutes, StarRocks has its own internal guardrails: *

stream_load_default_timeout_second

: Default is 600s (10 mins). If your 100MB+ batch takes longer than 10 minutes to process (including the actual write to disk), StarRocks will kill the transaction. * Check/Fix: You can increase this globally in

fe.conf

or pass a custom timeout in the NiFi header:

timeout: 1200

. *

streaming_load_rpc_max_alive_time_sec

: This is a BE config (default 1200s). If your BEs are struggling to sync data across the 6 nodes, this RPC might time out. 5. NiFi

InvokeHTTP

Redirect Handling Stream Load works by hitting an FE, which then sends an HTTP 307 Redirect to a specific BE. * Requirement: Ensure

InvokeHTTP

is set to "Follow Redirects: true". * Trust: In a

curl

command, we use

--location-trusted

. In NiFi, "Follow Redirects" handles the location, but ensure your credentials are sent in the initial request headers (Basic Auth). Troubleshooting Steps: 1. Reduce Batch Size: Temporarily drop your NiFi

MergeContent

size to 10MB. If the error disappears, it’s a timeout/buffering issue. 2. Add Headers: Add

Expect: 100-continue

and

ignore_json_size: true

(if using JSON) to your NiFi processor. 3. Direct Hit: Try pointing NiFi directly at a BE IP (port 8040) instead of the LB/Nginx. If it works, the issue is definitely in your Nginx/LB configuration. References • faq/loading/Stream_load_faq.md • loading/StreamLoad.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md