< Rocky> I am seeing this error pretty frequently in my fe w StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> I am seeing this error pretty frequ...

GregM

05/12/2026, 2:18 AM

@Rocky I am seeing this error pretty frequently in my fe.warn log

2026-05-12 02:16:30.855Z WARN (publish-task-762|5800) [TransactionGraph.remove():127] remove txn 33596742 with dependency: [33596750] this may happen during FE upgrading

Rocky

05/12/2026, 2:18 AM

This warning message is related to the Shared-data (Cloud Native) architecture in StarRocks, specifically within the Batch Publish mechanism introduced in version 3.2.x. Why this happens The

TransactionGraph

is an internal component that manages dependencies between concurrent transactions in a shared-data cluster. Its job is to ensure that transactions are "published" (made visible) in the correct version order, even when multiple transactions are grouped together to improve performance. The warning

remove txn [ID_A] with dependency: [ID_B]

occurs when: * Transaction A is being removed from the graph (either because it finished publishing or was aborted). * The graph still shows that transaction B is depending on A. Is it a problem? In most cases, this warning is benign and can be ignored if your cluster is operating normally. * During FE Upgrades: The log itself notes "this may happen during FE upgrading." This is because during a restart or upgrade, the FE replays the edit log and reconstructs the transaction state. The graph might temporarily hold inconsistent dependency pointers until the replay is complete. * During Normal Operation: If you see this frequently without an upgrade, it is usually a sign of high-concurrency ingestion. Under heavy load, the cleanup of the transaction graph might lag slightly behind the actual publish tasks, leading to these "stale" dependency warnings when a transaction is finally purged from memory. When to be concerned You should only investigate further if this log is accompanied by: 1. Stuck Loads: Data ingestion is successful, but the data does not become visible (status stays

COMMITTED

but not

VISIBLE

). 2. FE Performance issues: High CPU/Memory usage on the FE specifically tied to the

publish-task

threads. 3. Transaction timeouts: Frequent

ABORTED

transactions with "publish timeout" reasons. Potential Mitigations If the frequency of these logs is cluttering your logs and you are on a high-concurrency workload, you can tune the batch publish settings in the FE configuration: * lake_batch_publish_max_version_num
: (Default 10) Lowering this may reduce the complexity of the transaction graph but might decrease ingestion throughput. * lake_publish_version_max_threads
: (Default 512) Ensure there are enough threads to handle the publish tasks. Recommendation: If there are no functional issues with data visibility or ingestion, you can safely ignore these warnings. If you are running an older version (e.g., early 3.2.x), upgrading to the latest maintenance release (like 3.2.10+ or 3.3.x) often includes optimizations that reduce these race-noise in these logs. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • administration/management/compaction.md

GregM

05/12/2026, 2:22 AM

@Rocky how can i check FE cpu

Rocky

05/12/2026, 2:23 AM

To check the CPU usage of a StarRocks FE (Front End) node, you can use several methods depending on whether you want real-time monitoring or historical data. 1. Operating System Level (Real-time) The most direct way to check FE CPU usage is using standard Linux system tools on the machine where the FE process is running: * top
: Run

top -Hp <fe_pid>

to see CPU usage by thread within the FE process. * htop
: Provides a more user-friendly visual interface for the same data. 2. FE Web UI (v3.2.0+) If you have access to the FE Web UI (usually

http://<fe_ip>:<http_port>

), you can view hardware information directly: * Check the Index page (

/index

). * In newer versions, there is a Hardware section that lists CPU, memory, and disk usage. _ _Note:* This requires the FE configuration

http_web_page_display_hardware

to be set to

true

(default). 3. Monitoring (Prometheus & Grafana) While StarRocks exports many internal metrics, FE CPU utilization at the OS level is typically monitored using a Prometheus Node Exporter on the FE host. * In a standard StarRocks Grafana dashboard, look for the System Load or Host sections. * For query-specific CPU consumption, check the

cpucostns

field in the fe.audit.log
, which records the CPU time (in nanoseconds) spent by the FE for each request. 4. Internal Profiling (v3.2.12+) If you are seeing high CPU and want to know what the FE is doing (e.g., which Java threads or functions are consuming it), StarRocks can generate CPU flame graphs: * Enable Profiler: Set

proc_profile_cpu_enable = true

fe.conf

(or via

ADMIN SET FRONTEND CONFIG

). * View Reports: The FE will use

AsyncProfiler

to generate HTML reports under

fe/log/proc_profile/

. These reports show exactly which code paths are using the CPU. 5. Check FE Metrics Endpoint You can manually curl the metrics endpoint to see available JVM and system metrics:

Copy code

bash
curl http://<fe_ip>:<http_port>/metrics

Look for

jvm_

related metrics or

starrocks_fe_

metrics, though note that for raw host CPU, system-level monitoring (Node Exporter) is the standard recommendation. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md • administration/management/monitoring/metrics.md References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md

GregM

05/12/2026, 2:27 AM

@Rocky i made these changes

Copy code

-- made a few config changes for the fe in custom-values.yaml and redeployed
-- changes were
-- default was 30min -> 24 hours
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';

-- default was 5 sec -> 30
ADMIN SHOW FRONTEND CONFIG LIKE 'starmgr_grpc_timeout_seconds';

-- default was true -> false
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_enable_batch_publish_version';

and now a jdbc insert process from nifi seems to be running slower, any reason why?

Rocky

05/12/2026, 2:27 AM

The reason your JDBC insert process is running slower is almost certainly due to disabling lake_enable_batch_publish_version
. In a StarRocks shared-data (Cloud Native) cluster, the "Publish" phase is the final step of a transaction where metadata is updated to make data visible. 1. The Impact of
lake_enable_batch_publish_version
(Most Likely Cause) When this is

true

(the default), StarRocks groups multiple concurrent transactions together into a single "batch publish" operation. This drastically reduces the number of RPC calls to the internal metadata service (StarMgr) and the object storage. * Why JDBC inserts feel it: JDBC inserts from tools like NiFi often result in many small, frequent transactions. * With Batching (
true
): Multiple JDBC inserts are committed in one go. * Without Batching (
false
): Each individual JDBC

INSERT

must wait for its own dedicated publish round-trip. In shared-data mode, this adds significant latency to every single commit. * Performance Difference: Official benchmarks show that under high concurrency, disabling this can drop ingestion throughput by up to 8x (e.g., from ~85MB/s down to ~10MB/s). Recommendation: Revert

lake_enable_batch_publish_version

true

. ────────── 2. Analysis of the Other Changes The other two changes are unlikely to cause a direct slowdown in ingestion throughput: * starmgr_grpc_timeout_seconds
(5s → 30s): This simply allows the FE to wait longer before timing out on internal metadata calls. It won't make a healthy system slower, but if the system is already slow (due to the batching issue above), this might cause your NiFi threads to hang for 30 seconds instead of failing quickly when the FE is overloaded. * lake_autovacuum_grace_period_minutes
(30m → 24h): This determines how long deleted or compacted data files are kept on S3/GCS before being deleted. Increasing this to 24 hours will increase your storage costs (as old versions aren't cleaned up for a day), but it has no impact on the speed of data being written. Summary Recommendation 1. Re-enable batch publish:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");

2. Keep the vacuum grace period short (unless you have specific requirements for very long-running queries or point-in-time recovery), as 24 hours can lead to significant storage bloat on high-churn tables. 3. Check FE Logs: If it's still slow after reverting, check the

fe.warn.log

for "publish timeout" or "slow rpc" messages. References • knowledge/usage/load/shared_data_batch_ingestion.md • administration/management/FE_configuration.md • faq/shared_data_faq.md

GregM

05/12/2026, 2:29 AM

@Rocky I turned

lake_enable_batch_publish_version

to false at your recommendation because it could be buggy and causing some of stale .dat references I was running into

Rocky

05/12/2026, 2:29 AM

I understand the conflict. To clarify, while turning off

lake_enable_batch_publish_version

resolves that specific "dependency" warning and simplifies the transaction logic, it is also the direct cause of the significant slowdown you are seeing in NiFi. Here is how to resolve the performance issue while still protecting against the "stale

.dat

references" (File Not Found errors) you encountered: 1. Fix the "Stale .dat References" The "stale

.dat

references" usually happen when the Autovacuum process deletes data files that a slow-running query or a late-arriving publish task still needs. * Your Change was Correct: Changing

lake_autovacuum_grace_period_minutes

from 30m to 24 hours is the standard and correct way to fix this. It ensures that files are kept on the object store for a full day after they are no longer needed, providing a massive safety buffer. * Result: This change alone likely fixed your

.dat

reference errors. You do not need to disable Batch Publish to fix this specific issue. 2. The Slowdown (The "Buggy" Batch Publish) The

lake_enable_batch_publish_version

feature was indeed "noisy" (causing many warnings) and had some edge-case bugs in early 3.2.x releases. However, for NiFi/JDBC (which creates many small transactions), this feature is critical for performance. * The Problem: Without it, every single NiFi

INSERT

must wait for a full round-trip to the metadata service. * The Fix: 1. Re-enable Batch Publish:

ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");

2. Upgrade (Recommended): The specific bug causing those "remove txn with dependency" warnings and related metadata inconsistencies was fixed in later patches. If you are on an early 3.2.x version, upgrading to 3.2.10 or higher (or 3.3.x) allows you to keep Batch Publish on safely. 3. Alternative "Middle Ground" If you cannot upgrade immediately and are worried about Batch Publish bugs, but need performance, try this: 1. Re-enable Batch Publish (

lake_enable_batch_publish_version = true

). 2. Reduce the Batch Size:

Copy code

sql
    -- Reduce the number of versions batched together (default is 10)
    ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "5");

This provides a balance: it still batches your NiFi inserts to keep it fast, but it reduces the complexity of the transaction graph that was causing the warnings. Summary of Recommendation * Keep

lake_autovacuum_grace_period_minutes = 1440

(24h) to prevent the "stale reference" errors. * Turn
lake_enable_batch_publish_version
back to
true
to fix your NiFi performance. * If you still see the

WARN

logs about "dependency," they are benign in newer patches and can be ignored. They do not cause data loss or stale references. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • administration/management/compaction.md • faq/shared_data_faq.md

GregM

05/12/2026, 2:34 AM

@Rocky what's a reasonable CPU % for an fe pod

Rocky

05/12/2026, 2:34 AM

For a StarRocks FE pod, "reasonable" CPU usage depends on your cluster's activity, but here are the general guidelines for production: 1. Typical Usage Ranges * Idle / Steady State: 5% – 20%. In a healthy cluster, the FE should not be working very hard. It primarily manages metadata (BDBJE) and handles heartbeat signals. * High Load / Peak Traffic: 40% – 70%. When NiFi or other tools are sending high-concurrency

INSERT

statements or complex queries, the FE CPU will spike because it has to plan every query and manage transaction commits. * Warning Zone: > 80% (Sustained). If CPU stays above 80% for long periods, you will likely see "Query Planning" latency increase, and the FE may struggle to maintain the Raft quorum for metadata, potentially leading to cluster instability. 2. What Consumes CPU in the FE? Since the FE is a Java process, its CPU usage is driven by: * Query Planning: Parsing SQL, analyzing metadata, and generating execution plans. * Transaction Management: In shared-data (Cloud Native) mode, the FE is heavily involved in coordinating commits to the metadata service. * Metadata Synchronization: Keeping the Follower FEs in sync with the Leader via BDBJE. * JVM Garbage Collection (GC): If memory is tight, high CPU is often a side effect of the JVM constantly trying to free up heap space. 3. How to Interpret Your Current Slowdown If your NiFi process is running slower after your recent config changes: * If FE CPU is LOW (< 30%): The bottleneck is latency. By disabling

lake_enable_batch_publish_version

, every NiFi insert is waiting for a dedicated metadata trip. The FE isn't "busy," it's just making NiFi wait for a response. * If FE CPU is HIGH (> 80%): You are likely experiencing metadata contention. Turning off batching means the FE has to process many more individual transaction requests per second. This generates massive overhead for the FE's internal lock manager and metadata logs. Recommendations 1. Check for "Stealing": In Kubernetes, ensure your FE pod has CPU Requests equal to its Limits (Guaranteed QoS). Java applications perform poorly if the underlying K8s node "throttles" the CPU. 2. Monitor the Leader: Only the Leader FE handles the heavy lifting for writes. Check the CPU specifically on the Leader node. 3. Audit Logs: Look at the

fe.audit.log

and check the

queryTime

vs.

planTime

. If

planTime

is high (hundreds of ms), your FE is CPU-bound. Summary: Aim to keep your FE pods below 70% during peak bursts to ensure there is enough "headroom" for the metadata service to remain responsive. If you are consistently above that, consider scaling the FE vertically (adding more cores) or re-enabling Batch Publish.

GregM

05/12/2026, 2:39 AM

@Rocky I turned the batching back on and immediately got

Copy code

2026-05-12 02:36:46.167Z ERROR (publish-task-580|3685) [PublishVersionDaemon.publishPartitionBatch():601] publish partition batch partition.getVisibleVersion() + 1 != version.get(0) 32988495 75807 75807

i also saw my cpu spiking a pretty good bit on my fe leader

Rocky

05/12/2026, 2:39 AM

This error and the CPU spike confirm that your cluster is hitting a known version-discontinuity bug in the Batch Publish logic for shared-data (Cloud Native) tables. 1. What the Error Means

publish partition batch partition.getVisibleVersion() + 1 != version.get(0)

The FE is trying to commit a "batch" of transactions (likely those many small inserts from NiFi). * It expected the next data version to be, for example,

(

getVisibleVersion() + 1

). * However, the batch of transactions it prepared starts at version

(

version.get(0)

). * The Result: There is a "hole" (version 101) in the sequence. StarRocks refuses to publish the batch to prevent data inconsistency, causing the task to fail and retry. 2. Why the FE CPU is Spiking The CPU spike on the Leader FE is caused by Transaction Graph Overhead: * With many small JDBC inserts and Batching enabled, the FE builds a "dependency graph" to group these transactions. * When a "hole" or "gap" occurs (the error above), the FE spends significant CPU cycles repeatedly trying to re-calculate the batch, managing locks, and cleaning up failed publish tasks. * Basically, the FE is "spinning" trying to reconcile the metadata for hundreds of tiny transactions that it can't find a continuous sequence for. 3. How to Resolve This You are caught between a performance bottleneck (batching OFF) and a metadata bug (batching ON). Option A: The Long-Term Fix (Upgrade) This specific logic (how the FE handles version gaps during batching) was significantly improved in later patches. * If you are on an early 3.2.x release, upgrade to 3.2.10+ or 3.3.x. * Users on 3.3.x and 3.4.x rarely see this specific "partition version + 1" error as the transaction manager became much more robust. Option B: The "Stabilize Now" Fix (Tuning) If you can't upgrade immediately, try to make the "batches" smaller so the FE is less likely to hit a gap: 1. Keep

lake_enable_batch_publish_version

set to

true

. 2. Reduce the batch size significantly:

Copy code

sql
    -- Default is 10. Lowering this to 2 or 3 reduces the chance of a version gap
    -- while still providing much better performance than "false" (which is 1).
    ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "3");

3. Increase NiFi Batching: If possible, configure your NiFi JDBC sink to commit larger batches of rows (e.g., 1000–5000 rows per

INSERT

) rather than many tiny inserts. Fewer transactions = less work for the FE metadata manager. Summary The CPU spike and error are symptoms of the FE struggling to sequence your NiFi transactions. * Reverting to
false
stops the error and CPU spike but kills your NiFi throughput. * Lowering
lake_batch_publish_max_version_num
is your best middle ground to keep things moving while you plan an upgrade to upgrade to a version with more stable Shared-Data metadata handling. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/loading/Loading_faq.md

GregM

05/12/2026, 3:28 AM

@Rocky how about this error

Copy code

2026-05-12 03:28:00.497Z ERROR (starrocks-mysql-nio I/O-2|141) [MySQLReadListener.handleEvent():97] Unexpected error in MySQLReadListener
 java.lang.IllegalStateException: receive packet sequence id[7] expected[0]

Rocky

05/12/2026, 3:28 AM

This error (

IllegalStateException: receive packet sequence id[7] expected[0]

) is a MySQL protocol desynchronization error. It is a direct secondary symptom of the FE CPU spike and the Batch Publish error you just saw. Here is the chain of events: 1. Metadata Collision: Your NiFi process is sending many concurrent inserts. With batching enabled, the FE tries to group them. Because of the version-discontinuity bug (the

partition batch

error), the FE enters a heavy retry loop. 2. CPU Spike & Latency: This retry loop spikes the FE Leader's CPU. The FE becomes so busy managing metadata that it stops responding to the network (the MySQL NIO thread) in a timely manner. 3. Client Timeout / Retry: NiFi or your JDBC driver sees the "hang" and likely attempts to reset the connection, time out, or send a new command on the same socket. 4. Protocol Mismatch: When the FE finally "wakes up" to read the network buffer, it expects a brand new command (which must start with Sequence ID 0). Instead, it finds a packet from the client that has Sequence ID 7 (part of an older or failed exchange). The FE doesn't know how to handle a mid-sequence packet when it's expecting a fresh start, so it throws this

IllegalStateException

. How to Fix You do not need to debug the MySQL error directly; it will go away once you stabilize the FE's metadata performance. Immediate Mitigation: 1. Lower the Batch Limit: To stop the CPU spike and the "version + 1" errors, restrict how many transactions the FE tries to bundle at once:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "2");

Why? Setting this to

(default is 10) still gives you the performance benefit of batching but drastically reduces the mathematical complexity and the likelihood of hitting a "version hole" that triggers the FE's CPU-heavy retry logic. 2. Increase NiFi Batch Size: If you are using the

PutDatabaseRecord

processor in NiFi, increase the Record Tabular Data Batch Size. It is better for StarRocks to receive one

INSERT

with 1,000 rows than 1,000 `INSERT`s with 1 row. 3. Check FE Version: If you are on a version earlier than 3.2.10, this batching logic is known to be fragile. Upgrading to a later patch of 3.2 or to 3.3.x includes significant fixes for how the FE handles these "holes" in the transaction sequence without spiking CPU or crashing the MySQL listener. Summary: The MySQL error is just a sign that your FE is "choking" on the metadata tasks. Fix the batching bottleneck using the

lake_batch_publish_max_version_num

setting, and the protocol errors will disappear. References • loading/Json_loading.md • administration/Meta_recovery.md

GregM

05/12/2026, 3:36 AM

@Rocky is the stream load better than a jdbc load?

Rocky

05/12/2026, 3:37 AM

Yes, Stream Load is significantly better than JDBC loads for high-throughput or real-time data ingestion in StarRocks. The issues you are currently seeing—FE CPU spikes, version discontinuity errors, and MySQL protocol desyncs—are almost always caused by using JDBC for frequent, high-concurrency inserts. Why Stream Load is Better Feature | JDBC (
INSERT INTO
) | Stream Load (HTTP) Protocol | MySQL Wire Protocol (Overhead) | HTTP (Lightweight) FE Overhead | High: Every statement is parsed & planned | Low: Bypasses SQL parsing; data goes direct to BE Transaction Cost | High: 1 Transaction per

INSERT

| Low: 1 Transaction per batch (up to GBs) Concurrency | Limited: FE bottlenecked by SQL locks | High: Designed for parallel streaming Data Format | Row-based SQL strings | Efficient CSV or JSON The Problem with JDBC in your NiFi setup When you use a JDBC Sink (like NiFi’s

PutDatabaseRecord

), each "batch" of rows is typically sent as an

INSERT

statement. * If NiFi is sending these every second or two, the StarRocks FE (Leader) has to generate a new transaction ID and a new "version" for every single hit. * In shared-data (Cloud Native) mode, this creates massive metadata contention. * The

IllegalStateException

(sequence ID mismatch) happens because the FE is so busy processing these tiny SQL transactions that the MySQL connection gets out of sync. Recommendations for NiFi Users 1. If you stay with JDBC (Short term) You must batch more aggressively in NiFi. * In your

PutDatabaseRecord

(or similar) processor, set the "Statement Batch Size" to a large number (e.g., 5,000 or 10,000). * Increase the "Batch Size" and "Max Binage Size" in the NiFi controller service. _ _Goal:* Reduce the number of transactions per second. 2. The "StarRocks Way" (Long term) The best practice for NiFi is to move away from JDBC and use Stream Load via the InvokeHTTP
processor (or a custom StarRocks NiFi NAR if available in your organization). * How it works: NiFi packages the data as a CSV or JSON flowfile and then sends a

PUT

request to

http://<FE_IP>:<FE_HTTP_PORT>/api/<db>/<table>/_stream_load

. * Result: The FE CPU usage will drop dramatically, and you will no longer see "sequence ID" or "version + 1" errors because you are committing data in larger, more efficient chunks. Summary If you are seeing FE CPU spikes, JDBC is likely the culprit. Switching to Stream Load (or significantly increasing your JDBC batch sizes) is the standard architectural fix to stabilize the cluster. References: • Loading Concepts: Stream Load vs INSERT • Stream Load API Reference References • loading/Spark-connector-starrocks.md • loading/StreamLoad.md • unloading/arrow_flight.md • loading/Loading_intro.md • unloading/Spark_connector.md

GregM

05/12/2026, 3:40 AM

@Rocky how big should i batch the json for stream loads?

Rocky

05/12/2026, 3:41 AM

For optimal performance and to avoid the FE overhead you've been seeing, you should aim for the following batch sizes when using Stream Load with JSON: 1. Recommended Size: 100 MB to 1 GB * The Sweet Spot: Aim for 100 MB to 500 MB per batch. * Why: StarRocks is a columnar database. It performs best when it can write "large" chunks of data at once. Small batches (e.g., 1 MB) create too many "versions" (small files), which leads back to the metadata pressure and CPU spikes you experienced. * The "Safety" Limit: By default, StarRocks has a limit of 100 MB for JSON Stream Loads to prevent high memory usage during parsing. * If your batch is larger than 100 MB, you must add this header to your request:

"ignore_json_size: true"

* The absolute maximum for a single Stream Load is typically 10 GB, but for JSON, anything over 1–2 GB starts to become very CPU-heavy to parse. 2. Frequency: Every 5 to 30 Seconds Instead of sending data as soon as it arrives, configure NiFi to "buffer" data for at least 5–10 seconds (or until it hits the ~100MB+ mark). * High Frequency = High Metadata Load: Even with Stream Load, sending 10 requests per second will eventually overwhelm the FE's ability to commit versions. * Low Frequency = Better Compression: Larger batches allow StarRocks to compress the data better on disk, leading to faster queries. 3. Essential Stream Load Headers for JSON When setting up your

InvokeHTTP

or Stream Load client in NiFi, ensure these headers are set: *

format: json

strip_outer_array: true

(if your NiFi output is a JSON array

[...]

) *

ignore_json_size: true

(if your batches might exceed 100 MB) *

Expect: 100-continue

(helps prevent unnecessary data transfer if the FE is busy) Summary Comparison Load Type | Size Recommendation | Why? JDBC (Current) | Small (Rows) | Bottlenecked by FE SQL parsing & locks. Stream Load | 100 MB - 1 GB | Bypasses SQL layer; optimal for disk I/O and compression. Pro-tip for NiFi: If you are using the

MergeContent

processor before your Stream Load, set the "Maximum Number of Bins" and "Max Group Size" to reach that 100MB+ target before triggering the load. References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/Json_loading.md • loading/StreamLoad.md

GregM

05/12/2026, 3:45 AM

@Rocky can the stream endpoint be the cluster load balancer?

Rocky

05/12/2026, 3:45 AM

Yes, the stream endpoint can and should be the cluster Load Balancer (LB). However, there is a critical technical detail you must handle for this to work correctly: HTTP Redirection. How the Stream Load flow works: 1. Request to LB: You send the Stream Load request to your Load Balancer (e.g., F5, Nginx, or AWS ELB) on the FE HTTP port (default

). 2. LB to FE: The LB forwards the request to one of your StarRocks FEs. 3. FE Redirect (The "Catch"): The FE does not actually "swallow" the data. It selects a Backend (BE) node to coordinate the load and sends an HTTP 307 Redirect back to the client, pointing directly to that BE's IP and port (default

). 4. Client to BE: The client must then follow that redirect to send the actual data payload to the BE. Important Configuration for your Load Balancer For a smooth experience through an LB, keep these three things in mind: 1. Enable "Follow Redirects" in your Client If you are using NiFi (

InvokeHTTP

) or

curl

, you must ensure the client is configured to follow redirects. * NiFi: Set the "Follow Redirects" property in the

InvokeHTTP

processor to

True

. * curl: Use the

-L

flag. 2. Network Path to BEs The client (NiFi) must have a direct network path to the BE IP addresses. * If your LB is the only thing accessible to NiFi, and the BEs are on a private network that NiFi cannot reach, the redirect will fail (NiFi will try to connect to a private BE IP and timeout). * The Fix: If the BEs are not directly reachable, you must enable the FE Proxy feature. This tells the FE to proxy the data itself rather than redirecting the client. * In FE config (

fe.conf

enable_fe_proxy = true

_ _Note:* Using the proxy is slightly less efficient but is often necessary for strict network environments where only the LB is exposed. 3. Health Checks Configure your Load Balancer to check the health of the FEs using the following endpoint: *

GET http://<FE_IP>:8030/api/bootstrap

* A

200 OK

response indicates the FE is up and part of the cluster. Summary Using a Load Balancer for the endpoint is the recommended production setup because it ensures high availability. Just ensure your NiFi processor is set to Follow Redirects and that there is connectivity between NiFi and the BE nodes (or enable

enable_fe_proxy

). References • loading/StreamLoad.md • quick_start/helm.md • administration/management/resource_management/Load_balance.md • administration/management/FE_configuration.md • faq/Deploy_faq.md References • quick_start/helm.md • administration/management/resource_management/Load_balance.md • administration/management/FE_configuration.md • loading/StreamLoad.md • faq/Deploy_faq.md

GregM

05/12/2026, 3:49 AM

@Rocky can i use dynamic partitions with stream load? by that i mean if the target table DDL is such that a column is used for a partition, will that be respected with the stream load data?

Rocky

05/12/2026, 3:49 AM

Yes, Stream Load fully respects your table's partitioning DDL. You do not need to manually specify the partition names in your load headers for this to work. Depending on how you defined your table, the behavior varies slightly: 1. Automatic Partitioning (Recommended, v3.1+) If you used the newer Expression Partitioning syntax (e.g.,

PARTITION BY dt

PARTITION BY date_trunc('day', ts)

): * Automatic Routing: StarRocks will automatically look at the value in your data's partition column and route it to the correct partition. * Automatic Creation: If a partition for that specific value (e.g., a new day) does not exist yet, StarRocks will create it on-the-fly during the Stream Load. * No Headers Needed: You do not need to add a

partitions

header to your request. 2. Legacy Dynamic Partitioning If you used the older properties style (

"dynamic_partition.enable" = "true"

with

PARTITION BY RANGE

): * Automatic Routing: Data is still automatically routed based on the column value. * Strict Window: StarRocks only creates partitions in advance based on your

dynamic_partition.end

setting. If you try to Stream Load data for a date that falls outside that pre-created window, the load will fail with an error (e.g., "no partition found"). * No Headers Needed: Just like the newer method, you don't need to name the partitions in your HTTP request. 3. Manual Partitions If you created partitions manually (without dynamic properties): * Automatic Routing: Data is routed correctly if the partition already exists. * Failure on Missing: If the data value doesn't match an existing partition, the load will fail. Summary of Best Practices * Don't use the
partitions
header: In 99% of cases, you should leave the

partitions

header out of your Stream Load. The system is designed to handle routing automatically based on your table schema. * Use v3.1+ Expression Partitioning: This is the most "hands-off" approach for NiFi or any streaming source, as it eliminates the need to manage partitions or worry about "out of range" errors. Example Stream Load for a partitioned table:

Copy code

bash
# Even if 'dt' is the partition column, you don't need to specify it here.
# StarRocks reads 'dt' from the JSON/CSV and handles the rest.
curl --location-trusted -u user:pass \
    -H "format: json" \
    -T data.json \
    <http://fe_host:8030/api/db/my_partitioned_table/_stream_load>

References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • table_design/data_distribution/expression_partitioning.md • table_design/data_distribution/dynamic_partitioning.md References • table_design/data_distribution/expression_partitioning.md • table_design/data_distribution/Data_distribution.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • table_design/data_distribution/dynamic_partitioning.md

GregM

05/12/2026, 1:17 PM

@Rocky i'm trying to test out the stream load and getting this message after sending the request to my loadbalancer at 8030

Request Processing failed: FlowFile[filename=9efc8559-8a63-4279-9749-5a8a59663bad]: java.net.UnknownHostException: kube-starrocks-cn-1.kube-starrocks-cn-search.kube-starrocks.svc.cluster.local: Name or service not known

Rocky

05/12/2026, 1:17 PM

This error is happening because of the HTTP 307 Redirect behavior I mentioned earlier. The FE is redirecting your NiFi request to a specific internal Kubernetes hostname (

kube-starrocks-cn-1...

). Since NiFi is likely running outside of that Kubernetes internal network (or lacks the K8s DNS resolver), it doesn't know how to reach that address, resulting in the

UnknownHostException

. To fix this in a Kubernetes environment, you have two primary options: Option 1: Enable FE Proxy (Easiest) You can tell the StarRocks Frontend (FE) to proxy the data itself instead of redirecting the client. This way, NiFi only ever talks to the Load Balancer/FE, and never needs to resolve the internal BE/CN hostnames. 1. Update FE Configuration: Add the following to your

fe.conf

(or via the StarRocksCluster operator spec):

Copy code

properties
    enable_fe_proxy = true

2. How it works: When NiFi sends data to the LB, the FE will receive it and forward it to the BE nodes internally. NiFi stays connected to the FE the whole time. 3. Header: In your NiFi

InvokeHTTP

processor, you may need to add the header:

Expect: 100-continue

Option 2: Deploy the StarRocks FE Proxy Component (Operator Recommended) If you are using the StarRocks Kubernetes Operator, there is a dedicated "FE Proxy" component (an Nginx-based service) specifically designed for this scenario. If you are using Helm, update your `values.yaml`:

Copy code

yaml
starrocks:
  starrocksFeProxySpec:
    enabled: true
    service:
      type: LoadBalancer # This creates an external IP for loading

You then point NiFi to this FE Proxy Load Balancer (port

) instead of the standard FE port. The proxy handles the internal redirection and hostname resolution for you. Quick Workaround (For testing only) If you just want to get the test moving and can't change configs yet: * In NiFi's

InvokeHTTP

processor, set Follow Redirects to

false

. _ _Note:* This will only work if the FE you hit happens to be the one coordinating the load, which isn't reliable for production. Recommendation: Use Option 1 (

enable_fe_proxy = true

) if you want to keep your current Load Balancer setup. It is the most straightforward way to solve network "line-of-sight" issues between a client and internal Kubernetes nodes inside a Kubernetes cluster. References • loading/StreamLoad.md • quick_start/helm.md • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/management/resource_management/Load_balance.md References • quick_start/helm.md • quick_start/shared-data.md • administration/management/resource_management/Load_balance.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • administration/Meta_recovery.md

GregM

05/12/2026, 1:23 PM

@Rocky I am using the operator but there are a fair amount of values for that feproxyspec, if i set enabled to true it will just work?

Rocky

05/12/2026, 1:23 PM

Yes, setting

enabled: true

and

service.type: LoadBalancer

is the primary way to solve this, and the operator handles the complex parts for you. When you enable the FE Proxy (

starrocksFeProxySpec

), the operator deploys a dedicated Nginx-based proxy that lives inside your Kubernetes cluster. This proxy is specifically designed to handle the "redirection" issue you encountered. How it "Just Works": 1. Image & Config: You don't need to specify an image or custom config; it defaults to a stable Nginx image and automatically generates the routing rules to find your FEs and BEs/CNs. 2. Resolution: Because the proxy lives inside the K8s network, it can resolve internal hostnames like

kube-starrocks-cn-1...

that your external NiFi instance cannot. 3. Client Experience: NiFi sends the data to the proxy, and the proxy manages the "307 Redirect" internally. To NiFi, it looks like a single successful request to one IP. The One Change You Must Make: The FE Proxy uses a different port than the standard FE. * Target Port: You must change your NiFi

InvokeHTTP

URL to use port 8080
(the default for the FE Proxy) instead of

. * LB IP: Use the External IP assigned to the new FE Proxy service (not the old FE service). Your updated
values.yaml
should look like this:

Copy code

yaml
starrocks:
  starrocksFeProxySpec:
    enabled: true
    service:
      type: LoadBalancer
    # Optional: If you have a specific IP you want to reuse
    # loadbalancerIP: "x.x.x.x"

Verification: After applying this, check that the proxy pod is running:

Copy code

bash
kubectl get pods | grep fe-proxy
kubectl get svc | grep fe-proxy  # Use this External IP and port 8080 in NiFi

Recommendation: Switch your NiFi endpoint to this new Load Balancer on port

. This will eliminate the

UnknownHostException

because NiFi will no longer be forced to resolve internal Kubernetes DNS names. References • quick_start/helm.md • loading/StreamLoad.md • administration/management/FE_configuration.md • faq/Deploy_faq.md References • quick_start/helm.md • administration/management/FE_configuration.md • loading/StreamLoad.md • faq/Deploy_faq.md • quick_start/shared-data.md References • deployment/environment_configurations.md • administration/management/FE_configuration.md • quick_start/helm.md • administration/stargo.md

GregM

05/12/2026, 1:27 PM

@Rocky how risky is that change to the rest of my cluster?

Rocky

05/12/2026, 1:28 PM

Enabling the

starRocksFeProxySpec

is very low risk and is the standard way to solve the issue you are seeing. Here is why it is safe to enable: 1. It is an Additive Component: The FE Proxy is a separate Nginx-based deployment. It does not replace or modify your existing FE or BE pods. Enabling it simply tells the operator to start a new set of small proxy pods alongside your cluster. 2. No Cluster Restart: In a standard Kubernetes Operator setup, adding a new spec (like the proxy) does not trigger a rolling restart of your database nodes (FE/BE/CN). Your current queries and data should continue running without interruption. 3. Isolation: The proxy only handles network routing. If the proxy pods were to fail or be deleted, it would only affect the external Stream Load traffic going through port 8080; the core StarRocks cluster and your SQL connections (via port 9030) would remain healthy. How to minimize risk while applying: To be 100% safe when you run

helm upgrade

, ensure that your

values.yaml

only contains the new proxy configuration and that you haven't accidentally changed any of the following for your existing FE/BE nodes: *

image

tags *

resources

(CPU/Memory) *

storageSpec

or volume names * The cluster name (

nameOverride

starrocksCluster.name

) Verification after the change: Once you apply the change, you can verify it is safe by running:

Copy code

bash
# Check that FE and BE pods have NOT restarted (look at the AGE column)
kubectl get pods -n <your-namespace>

# Check that the new proxy pods are running
kubectl get pods -n <your-namespace> | grep fe-proxy

Conclusion: It is a routine change designed specifically for your use case (external Stream Loading). It shouldn't cause any downtime for your existing StarRocks environment. References • quick_start/helm.md • faq/Deploy_faq.md • loading/StreamLoad.md • administration/management/resource_management/Load_balance.md References • faq/Deploy_faq.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • administration/management/graceful_exit.md • administration/management/FE_configuration.md

GregM

05/12/2026, 1:50 PM

@Rocky i'm getting

Copy code

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-starrocks-fe-proxy   0/0     0            0           16m

how can i troubleshoot why that isn't working

Rocky

05/12/2026, 1:54 PM

0/0

status in a Kubernetes Deployment (0 Ready / 0 Desired) typically means the desired replica count is set to zero. The StarRocks Operator manages this deployment based on your

StarRocksCluster

resource. Here is how to troubleshoot and fix it: 1. Check the Desired Replicas in the Custom Resource The operator might not have received a replica count, or it defaulted to 0. Check what the "Source of Truth" says:

Copy code

bash
kubectl get starrockscluster -o yaml | grep -A 10 starRocksFeProxySpec

Look for a

replicas:

field. If it is missing or set to

, the deployment will stay at

0/0

. Fix: Update your

values.yaml

(if using Helm) or your YAML manifest to explicitly include `replicas: 1`:

Copy code

yaml
starRocksFeProxySpec:
  enabled: true
  replicas: 1  # Add this line explicitly
  service:
    type: LoadBalancer

2. Check the Operator Logs If you have

replicas: 1

set but the deployment is still

0/0

, the StarRocks Operator might be hitting an error when trying to create the underlying resources (like a permission issue or a validation error). Find the operator pod and check its logs:

Copy code

bash
kubectl get pods -n <operator-namespace> | grep operator
kubectl logs <operator-pod-name> -n <operator-namespace>

Look for messages containing

fe-proxy

reconcile error

. 3. Check the StarRocksCluster Status The operator reports its progress in the "Status" section of the cluster resource. This often contains the exact reason why a component isn't starting:

Copy code

bash
kubectl get starrockscluster <cluster-name> -o jsonpath='{.status}'

Check if

FEPROXYSTATUS

says something other than

running

(e.g.,

failed

reconciling

). 4. Check for Resource Quotas In some environments, if you haven't specified

resources

(CPU/Memory) for the proxy, and your namespace has a ResourceQuota that requires limits, Kubernetes will refuse to create the pods. The Deployment will exist, but it won't be able to create a "ReplicaSet." * Check the ReplicaSet:

kubectl describe rs -l app=kube-starrocks-fe-proxy

* Look for events: If you see "FailedCreate" events, it will tell you exactly why (e.g., "must specify limits" or "quota exceeded"). Summary Checklist * Does