http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> i've removed some high volume JDBC ...
# questions-and-troubleshooting
g
@Rocky i've removed some high volume JDBC inserts in favor of stream loads but it seems my heap use has gone up, not down.
r
It is common for FE heap usage to increase after switching to high-frequency Stream Loads because the Frontend (FE) maintains metadata for every load transaction. While Stream Load is more efficient for the cluster overall, the transition from JDBC inserts (which have their own FE overhead) to many small Stream Loads often shifts the memory pressure toward the StreamLoadMgr in the FE. Here are the most likely causes and how to address them: 1. High-Frequency Task Metadata StarRocks keeps a record of every Stream Load task in the FE memory for auditing and state tracking. If you are performing a very high volume of small loads, these records can accumulate quickly. * The Culprit: The 
stream_load_task_keep_max_num
parameter defaults to 1000 tasks, and 
stream_load_task_keep_max_second
defaults to 3 days. * The Fix: Reduce these values in your 
fe.conf
(or via 
SET PROPERTY
) to trigger more aggressive cleanup:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_num" = "500");
    ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_second" = "3600"); -- Keep for 1 hour instead of 3 days
2. FE Acting as an HTTP Proxy When you send a Stream Load to the FE, it usually issues an HTTP 307 Redirect to a BE. However, if your client/network setup prevents following redirects, the FE may act as a proxy, streaming the data through its own memory. * The Fix: Ensure your client is sending data directly to the BE nodes or is correctly following the FE's 307 redirect. Proxying large volumes of data through the FE will significantly increase heap churn. 3. Transaction/Label Metadata Every load creates a transaction. If you are using unique labels for every small batch, the transaction manager in the FE must track these. * Recommendation: If possible, batch your data into fewer, larger Stream Loads (e.g., 100MB+ per load or once every few seconds) rather than many tiny "high volume" requests. This reduces the metadata footprint significantly. 4. Comparison to JDBC Inserts JDBC 
INSERT
statements are processed as SQL plans. While they also consume FE memory during parsing and planning, they don't leave behind the same "Task" history objects that the 
StreamLoadMgr
keeps for Stream Loads. How to verify To confirm what is occupying the heap, you can: 1. Check FE Memory Logs: If you are on v3.3.6+, check 
fe/log/proc_profile
for flame graphs. 2. Inspect Metadata: Run 
SHOW STREAM LOAD;
to see how many tasks are currently tracked. 3. Run Jmap: Use 
jmap -histo <fe_pid> | head -n 20
to see if 
StreamLoadTask
or related internal objects are top consumers. If the heap usage continues to grow without stabilizing, you may need to increase the FE JVM heap size (
-Xmx
) in 
fe.conf
to accommodate the higher metadata requirements of your ingestion pattern-especially if your cluster manages a high number of tablets. References • administration/management/FE_configuration.md • loading/StreamLoad.md
g
@Rocky will the merge commit still show filtered rows if exist?
r
No, the behavior of Merge Commit (introduced in v3.4.0) regarding filtered rows is significantly different and stricter than standard Stream Load: 1. Data Quality Issues Cause Total Failure In a standard Stream Load, you can use 
max_filter_ratio
to allow a certain percentage of rows with data quality issues (e.g., schema mismatches) to be ignored while the rest of the load succeeds. With Merge Commit, this tolerance is effectively removed: * Atomic Failure: If even one request in a merged transaction contains data quality issues (unqualified rows), the entire merged transaction fails. This means all concurrent Stream Loads within that merge window will fail together. * Result: You will not see a successful "Merge Commit" result showing a count of 
NumberFilteredRows
> 0. Instead, you will receive a 
Fail
status, and all data in that transaction will be rolled back. 2. Rows Filtered by 
WHERE
Clause Rows that are excluded via a 
WHERE
clause are considered "Unselected" rather than "Filtered" (data quality errors). * Synchronous Mode: The JSON response will still show 
NumberUnselectedRows
. These do not cause the transaction to fail. * Asynchronous Mode: Since the server returns a response immediately after receiving the data (before the merge window closes and processing finishes), the response will not show the number of unselected or filtered rows. You must check the task status later. 3. How to See Filtered Rows if a Failure Occurs If a Merge Commit transaction fails due to data quality issues, you can identify the problem rows using: * ErrorURL: The response for the failed load will provide an 
ErrorURL
(if 
max_error_number
allows logging) where you can view the specific rows that triggered the failure. * Information Schema: From v4.0+, you can query 
information_schema.loads
to view the status and metadata of Merge Commit-enabled tasks for better observability. Summary Table Feature | Standard Stream Load | Merge Commit (v3.4+) Filtered Rows (Data Quality) | Permitted via 
max_filter_ratio
| Causes failure of the entire merged batch Unselected Rows (WHERE) | Shown in response | Shown in response (Synchronous mode only) Response Timing | After processing finishes | Immediate (Async) or after commit (Sync) Recommendation: If your data stream frequently contains "dirty" records that you want to ignore, Merge Commit may not be suitable because it will cause high failure rates for all concurrent producers. In that case, either clean the data upstream or stick to standard (batch-optimized) standard Stream Loads. References • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • administration/management/BE_configuration.md • sql-reference/System_variable.md • table_design/indexes/Bitmap_index.md • loading/Flink-connector-starrocks.md
g
@Rocky retries are a bit more of a challenge with merge commit too since each label is unique, right?
r
You are correct. The shift to Merge Commit introduces a specific challenge regarding labels and idempotency that differs from standard Stream Load. 1. User Labels are Ignored In a standard Stream Load, your 
label
is the primary key for idempotency; if you retry with the same label, StarRocks prevents duplicate data. However, with Merge Commit (v3.4.0+), the system ignores user-specified labels for the merged transaction. The server automatically generates its own internal labels to manage the combined batch of requests. * The Problem: Because your label is ignored, StarRocks cannot perform its usual "Label Already Exists" check on a per-request basis. If your client times out and retries the same HTTP request, you may end up with duplicate data in the table. 2. Idempotency Shifts to the Table Model Since you lose label-based deduplication at the transaction level, you must handle idempotency differently: * Primary Key Tables: This is the recommended approach. Use a Primary Key table so that even if a retry causes a duplicate write, the row is simply updated/overwritten rather than duplicated. * Conditional Updates: In high-concurrency scenarios, you should also use 
merge_condition
(typically a timestamp or version column) to ensure that a retried "older" request doesn't overwrite a more recent successful write. 3. The "Fate Sharing" Problem Because Merge Commit bundles multiple requests into one transaction: * Atomic Failure: If any single request in the merged batch has a data quality issue (e.g., a schema mismatch), the entire batch fails. * Retry Complexity: When your request fails, it might not be due to your data. If you retry, you are essentially gambling that the next merged batch won't contain a different bad request from another client. Summary: Why it's a "Challenge" Feature | Standard Stream Load | Merge Commit Label Use | User-defined, enforced for uniqueness. | User-defined is **ignored**; system-generated. Retry Safety | Safe (Idempotent via Label). | Unsafe (Can cause duplicates). Failure Scope | Individual (Only your load fails). | Group (Your load fails if a neighbor's fails). Recommendation: If you move to Merge Commit, ensure your target table is a Primary Key table to handle the lack of label-based deduplication. If you need strict exactly-once semantics without a Primary Key table, standard Stream Load (with larger batches) is still the better choice. References • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/Flink-connector-starrocks.md • faq/loading/Loading_faq.md References • faq/loading/Loading_faq.md • loading/Flink-connector-starrocks.md • loading/StreamLoad.md
g
@Rocky i'm sending the stream load requests through the fe proxy, is that the correct way? i'm seeing my leader node heap % spike to 87%, is that ok?
r
The short answer is: Sending the request to the FE is the correct architectural pattern, but the FE should be redirecting the data, not proxying it. If your FE heap is spiking to 87%, you are likely hitting one of two bottlenecks: metadata accumulation or accidental data-proxying. 1. "Proxy" vs. "Redirect" (The Data Path) By default, when you send a Stream Load to an FE (port 8030), the FE performs a 307 Temporary Redirect. It tells your client, "Send this data to BE node X." * Correct behavior: The data flows from your client directly to the BE. The FE only handles the "handshake." * The Problem: If your client (e.g., a specific Java library or a misconfigured 
curl
) is configured not to follow redirects, or if you have a literal proxy (like Nginx or an ELB) in front of the FE that is "swallowing" the redirect and trying to stream the data through the FE, the FE will have to buffer and proxy those bytes. * Heap Impact: Proxying data through the FE JVM is extremely memory-intensive and will cause the exact heap spikes you are seeing. Check this: Ensure your client is following redirects (for 
curl
, use 
-L
). If you are using an external Load Balancer, it must be able to handle the 307 redirect from the FE or be configured to point to BEs directly for the data phase. 2. Why the Leader Node specifically? The FE Leader is responsible for the 
StreamLoadMgr
, which tracks the state of every transaction. * Metadata Overhead: Even if the data isn't flowing through the FE, the Leader must store metadata for every load (labels, status, timestamps). * High-Volume Inserts: If you replaced a few JDBC inserts with thousands of small Stream Loads, the Leader is now tracking thousands of individual tasks. * Retention: By default, StarRocks keeps this metadata for 3 days or up to 1000 tasks. If you are doing 1000 loads an hour, the Leader's heap will fill with 
StreamLoadTask
objects. 3. Is 87% Heap "OK"? No, 87% is dangerously high for a StarRocks FE. * 90% Threshold: Most StarRocks configurations trigger an alert at 90%. * Full GC Risk: Once you cross ~90%, the JVM will likely trigger a Full GC. Because the FE Leader's heap is often large, a Full GC can cause a "Stop the World" pause long enough to trigger a Leader Change (the cluster thinks the leader is dead). This will interrupt all current loads and queries. Immediate Mitigation Steps 1. Lower Metadata Retention: Trigger more frequent cleanup of old load tasks on the FE:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_num" = "500");
    ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_second" = "3600"); -- 1 hour
2. Verify Redirects: Check your client logs. Do you see HTTP 307 codes? If you see the FE node receiving the entire payload size in its own network metrics, it is proxying data instead of redirecting. 3. Increase FE Heap: If your metadata volume is simply high (many tablets + many loads), increase the 
-Xmx
in 
fe.conf
. For production clusters with high-frequency loading, 16GB or 32GB is a common requirement for the FE. References • loading/StreamLoad.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md • administration/Meta_recovery.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
g
@Rocky what are the risks of lowering those configs for task keeps
r
Lowering the retention and count limits for Stream Load tasks is a common and effective way to save FE heap memory, but it comes with several specific trade-offs. Here are the primary risks associated with reducing 
stream_load_task_keep_max_num
and `stream_load_task_keep_max_second`: 1. Loss of Idempotency (Duplicates) This is the most critical risk for standard Stream Loads (though less so for Merge Commit). StarRocks uses the 
label
to ensure "Exactly-Once" semantics. * The Risk: If you set 
label_keep_max_second
or the task count too low, StarRocks will "forget" that a specific label has already been processed. * The Result: If your client retries a load with an old label that was recently purged, StarRocks will treat it as a brand-new request and load the data again, causing duplicates. 2. Monitoring & Debugging Blindness These configurations control what you see in the FE's internal state and the StarRocks web UI. * The Risk: If a load fails at 2:00 PM and your retention is set to 30 minutes, by 2:31 PM you will have no record of that failure in 
SHOW LOAD
or 
information_schema.loads
. * The Result: Troubleshooting production issues becomes difficult because the history of 
ErrorURL
links (which point to why a load was filtered) will vanish. 3. Tooling & Connector Failures Many third-party tools and StarRocks' own connectors (like the Flink Connector) rely on polling the FE for the status of a load. * The Risk: If the Flink job is paused or a network hiccup occurs, and the retention period expires before the connector can check the status of a committed transaction, the connector may report a failure or be unable to recover. * The Result: You may see "Transaction not found" errors in your data pipeline logs, leading to manual intervention requirements. 4. High Cleanup Overhead While lowering the limits saves memory, setting them extremely low (e.g., keeping only a few dozen tasks) can backfire. * The Risk: The 
StreamLoadMgr
background thread will have to run more frequently and work harder to prune the 
idToStreamLoadTask
map. * The Result: You might trade high memory usage for increased FE CPU usage and garbage collection pressure from constant object destruction. Recommended "Safe" Minimums If your FE is at 87% heap, you should definitely lower these, but don't go to zero. * Retention: Keep at least 1 hour (
3600
) of history so you have time to react to alerts. * Count: Keep at least 500–1000 tasks. If you are doing hundreds of loads per minute, you may simply need to increase the FE Heap size (to 16GB or 32GB) rather than aggressive pruning. A Note on Merge Commit Since you are using Merge Commit, the risk of "Label Already Exists" (Risk #1) is already high because Merge Commit ignores your labels anyway. For Merge Commit users, Risk #2 (Observability) is your biggest concern, as you'll lose the ability to see the consolidated metadata for your merged batches. References • administration/management/FE_configuration.md • loading/loading_introduction/loading_considerations.md • faq/loading/Loading_faq.md • loading/Flink-connector-starrocks.md References • faq/loading/Loading_faq.md • administration/management/FE_configuration.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/Flink-connector-starrocks.md
g
@Rocky how can i see what my current number of stream load tasks is so i know what changing stream_load_task_keep_max_num will do
r
To see the current number of Stream Load tasks stored in your FE memory, the most accurate method is to query the internal metadata tables. 1. SQL Query (Recommended) You can count the entries directly from the 
information_schema
. This table reflects the tasks currently managed by the FE's 
StreamLoadMgr
.
Copy code
sql
-- Count only Stream Load tasks
SELECT count(*) FROM information_schema.stream_loads;

-- Alternatively, check all load types filtered by 'STREAM'
SELECT count(*) FROM information_schema.loads WHERE TYPE = 'STREAM';
Why this works: The rows in these views are essentially a window into the 
idToStreamLoadTask
map that 
stream_load_task_keep_max_num
controls. If this count is close to your current 
stream_load_task_keep_max_num
(default 1000), then StarRocks is actively pruning tasks, and increasing/decreasing this number will directly affect your FE heap usage. 2. FE Metrics Endpoint If you have monitoring set up (like Prometheus/Grafana), you can check the FE's 
/metrics
endpoint (usually 
http://<FE_IP>:8030/metrics
). Look for metrics such as: * 
starrocks_fe_stream_load_mgr_task_count
(if available in your version) * 
streaming_load_current_processing
(shows active loads, but not the total history). 3. Analyzing the Impact of the Config Change Before you change 
stream_load_task_keep_max_num
, check the count using the SQL above: * If the count is already at the limit: Your FE is already aggressively cleaning up tasks. Lowering the limit further will free up memory but will cause tasks to "disappear" from history even faster. * If the count is much lower than the limit: Then 
stream_load_task_keep_max_second
(the time-based limit) is likely the one doing the cleaning. Quick Tip on Heap Analysis Since your heap is at 87%, you can also check how many labels are being kept (which also consumes memory):
Copy code
sql
-- Check total labels across all load types
SHOW LOAD; -- Then look at the total row count returned
Recommendation: If you see thousands of rows in 
information_schema.stream_loads
, lowering 
stream_load_task_keep_max_num
to 500 is a safe first step to reduce memory pressure without losing too much debug visibility. References • sql-reference/information_schema/stream_loads.md • sql-reference/information_schema/loads.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md References • sql-reference/information_schema/stream_loads.md • sql-reference/information_schema/loads.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md References • sql-reference/information_schema/stream_loads.md • sql-reference/information_schema/loads.md • administration/management/monitoring/metrics.md
g
@Rocky i'm in shared-data arch, i shouldn't expect the same redirect as i would in shared-nothing, correct?
r
In shared-data (compute-storage separation) architecture, the redirection behavior is conceptually the same as in shared-nothing architecture, but there are a few infrastructure differences that might be causing your FE heap spikes. 1. Does the FE still redirect? Yes. Even in shared-data mode, StarRocks is designed to use an HTTP 307 redirect. * Shared-Nothing: FE redirects to a BE (Backend). * Shared-Data: FE redirects to a CN (Compute Node). If your client (or an external proxy like an ELB/Nginx) is not following that redirect, the data will be "proxied" through the FE's JVM. This is a common cause of 80%+ heap usage because the FE is forced to buffer incoming data it wasn't designed to handle. 2. Why the Leader FE is still a bottleneck in Shared-Data In shared-data clusters, the FE Leader actually has more work to do regarding metadata, which might explain the spike even if the data isn't being proxied: * Transaction Management: The FE Leader is the single point of coordination for all transactions. In shared-data, every load must interact with the object storage metadata (S3/GCS/OSS), and the FE manages the transaction commits and versioning. * Merge Commit (v3.4+): If you are using the new "Merge Commit" feature to handle high-concurrency small writes, the FE Leader performs the heavy lifting of grouping these requests. While this reduces the number of small files on S3, it increases the memory pressure on the FE Leader to track these pending merges. 3. Check for "Internal Proxying" In some cloud/K8s environments using the StarRocks Operator, users often point their Stream Load to a Load Balancer sitting in front of the FEs. * If that Load Balancer is configured to terminate the connection and doesn't allow the client to receive the 307 redirect pointing to the CN's internal IP/FQDN, the load will fail or be forced through the proxy. * FQDN Mode: In shared-data (especially K8s), ensure your FE is started with 
--host_type FQDN
. This ensures the redirect sent to your client uses a resolvable hostname (like 
sr-cn-0.sr-cluster...
) rather than an internal Pod IP that your client might not be able to reach. How to verify Run a manual load using 
curl -v -L
. * If you see 
< HTTP/1.1 307 Temporary Redirect
followed by a 
Location: http://<CN_IP>:8040...
, the redirect is working. * If the FE Leader heap continues to spike while the redirect is working, it is likely metadata volume. Check the number of active segments and tablet versions in your shared-data cluster. Summary You should expect a redirect. If the FE is acting as a proxy, it is usually because the client isn't following the 307 code or the network path (LB/WAF) is blocking it. If the heap is at 87% on a Leader node in shared-data, you likely need to increase the FE 
-Xmx
or tune the 
stream_load_task_keep
settings mentioned previously. References • loading/StreamLoad.md • quick_start/shared-data.md • administration/management/FE_configuration.md References • loading/StreamLoad.md • quick_start/shared-data.md • faq/shared_data_faq.md
g
@Rocky can you help me form a manual load?
r
To verify if your FE is correctly redirecting to a CN (Compute Node) and not proxying data through the JVM, you can run a manual load using 
curl
. The Manual Stream Load Command Run this from a machine that has access to your cluster. Replace the placeholders with your actual details:
Copy code
bash
curl -v --location-trusted -u 'your_user:your_password' \
    -H "label:manual_test_$(date +%s)" \
    -H "Expect:100-continue" \
    -H "column_separator:," \
    -T /path/to/your/data.csv \
    -XPUT http://<FE_LEADER_IP>:8030/api/<your_db>/<your_table>/_stream_load
What to look for in the output (
-v
) Because we added the 
-v
(verbose) flag, you can watch the "handshake" happen. This is how you confirm the redirect: 1. The First Request (to FE): You will see a request to the FE. Look for the response code: 
> PUT /api/db/table/_stream_load HTTP/1.1
< HTTP/1.1 307 Temporary Redirect
< Location: http://<CN_IP>:8040/api/_load_recording...
* If you see `307 Temporary Redirect`: The FE is doing its job. It is telling the client "don't give the data to me, give it to this CN." * If you see 
200 OK
immediately without a redirect: The client (curl) might not be showing the jump, or your load balancer is intercepting and proxying the request. 2. The Second Request (to CN): Because of 
--location-trusted
, curl will automatically follow that 
Location
header and send the actual file data (
-T
) to the CN's IP on port 8040. Key Flags Explained * 
--location-trusted
: Required. This tells curl to follow the 307 redirect AND send your username/password to the new destination (the CN). Without "trusted", curl drops the credentials on the redirect, and the load will fail with 
401 Unauthorized
. * 
-H "Expect: 100-continue"
: Highly recommended for large files. It allows the CN to check if it's ready to receive data before curl starts pushing the whole file. * 
label
: Must be unique. Using 
$(date +%s)
ensures you don't get a "Label already exists" error during testing. Why this matters for your Heap issue If you run this command and it works, but your FE Heap is still at 87%, then the memory pressure is coming from Metadata (the thousands of labels/tasks we discussed) rather than the Data Transfer itself. If the command fails or hangs when trying to reach the CN IP, then your network/firewall is likely preventing the client from talking to the CNs directly. This forces users to use a proxy, which is exactly what kills FE memory. References • loading/StreamLoad.md • quick_start/shared-data.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • administration/management/monitoring/metrics.md References • quick_start/shared-data.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md
g
@Rocky does it matter if i have real data? would it even make it that far before showing redirect or not?
r
It does not matter if you have real data. You can test this with a single line of dummy text. The redirect happens at the HTTP protocol level before the actual data transmission begins. Here is how the sequence works: 1. Request: Your client (curl) sends the HTTP headers to the FE. 2. Decision: The FE checks if the database and table exist and selects a healthy CN/BE to coordinate the load. 3. Redirect: The FE sends back an 
HTTP 307 Temporary Redirect
containing the IP of the CN. 4. Data Transfer: Your client sees the 307, stops talking to the FE, and opens a new connection to that CN to actually upload the file. How to test with "Fake" data You can create a dummy file with just one row to see the redirect:
Copy code
bash
echo "1,test_data,2023-01-01" > dummy.csv

curl -v --location-trusted -u 'user:pass' \
    -H "label:test_$(date +%s)" \
    -T dummy.csv \
    -XPUT http://<FE_LEADER_IP>:8030/api/<db>/<table>/_stream_load
What you will see in the output: Even if the data itself is "bad" (e.g., column count doesn't match), you will see the redirect first: * Look for: 
> PUT /api/...
* Followed by: 
< HTTP/1.1 307 Temporary Redirect
_ Followed by: 
_ Issue another request to: 'http://<CN_IP>:8040/...'
Conclusion: If you see the 
307
and the 
CN_IP
in your 
curl -v
output, the redirect is working. If your FE heap is still high, it confirms the issue is metadata storage (the volume of labels/tasks being kept in memory) rather than the data itself being proxied through the FE. References • loading/StreamLoad.md • quick_start/shared-data.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • administration/management/monitoring/metrics.md References • quick_start/shared-data.md • faq/shared_data_faq.md • loading/StreamLoad.md
g
@Rocky Interesting, I don't see that redirect, is that because i'm using the fe-proxy from the helm chart?
r
Yes, that is exactly why you aren't seeing the 
307 Redirect
in your client output. How 
fe-proxy
changes the behavior When you use the 
fe-proxy
 (which is a specialized Nginx sidecar/service provided in the StarRocks Helm chart), the flow changes to ensure external clients can load data without needing direct access to internal Kubernetes Pod IPs: 1. Client $\rightarrow$ `fe-proxy`: You send your 
Stream Load
request to the proxy (typically port 8080). 2. 
fe-proxy
$\rightarrow$ FE: The proxy forwards the headers to the FE Leader. 3. FE $\rightarrow$ `fe-proxy`: The FE sends the 
307 Redirect
back with the internal IP of a CN/BE. 4. 
fe-proxy
handles the jump: Instead of passing that 307 back to you (the client), the 
fe-proxy
(Nginx) follows the redirect internally. It opens a connection to the CN and starts streaming your data. 5. Result: To your client, it looks like a single 
200 OK
connection because the proxy handled the "redirection" logic behind the scenes. What this means for your FE Heap This is actually good news for your FE memory. * The Data Path: The actual data bytes are being "proxied" by the 
fe-proxy
container (Nginx/C++), not by the FE (Java). This means the data transfer is not consuming your FE JVM Heap. * The Conclusion: Since you are using the proxy and the data isn't touching the FE JVM, your 87% heap usage is definitely not caused by data proxying. Why is your FE Heap still high? If you are at 87% heap in a shared-data cluster while using the 
fe-proxy
, the pressure is coming from Metadata, not Data. In shared-data mode, the FE Leader is under higher pressure because: 1. Transaction Metadata: Every Stream Load creates a transaction. If you have a high frequency of small loads, the FE Leader must store all these transaction states and labels in memory. 2. Object Storage Coordination: The FE coordinates every write to S3/GCS. 3. Label Retention: By default, StarRocks keeps "load labels" for several days. If you are doing thousands of loads per hour, these millions of label strings sit in the FE Heap. Recommendation: Since you've confirmed you're using the proxy (and thus not proxying data through Java), you should focus on Metadata cleaning to lower your heap: * Shorten Label Expiration: Run 
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
(reduces retention to 1 hour instead of 3 days). * Check Stream Load Task Count: Run 
SHOW PROC '/statistic';
to see how many active/finished tasks are being tracked. * Scale Vertically: If your metadata volume is naturally high (many tablets/versions), you likely just need to increase the FE 
-Xmx
to 16GB or 32GB. References • loading/StreamLoad.md • quick_start/shared-data.md • administration/management/FE_configuration.md • faq/shared_data_faq.md References • loading/StreamLoad.md • quick_start/shared-data.md • faq/shared_data_faq.md • administration/management/FE_configuration.md
g
@Rocky can i be sure the redirect is working correctly with the fe-proxy? is there anything else I should do to verify?
r
You are correct: you don't see the redirect because the 
fe-proxy
is designed to "swallow" the 307 and handle the jump for you. Since the proxy is a C++ based Nginx process, the data is being moved at the network level and is not entering the FE's Java Heap. Here is how you can be 100% sure it’s working correctly: 1. Check the Stream Load Response JSON Every successful (or failed) Stream Load returns a JSON block. Look at the 
ErrorURL
field even if the load was a success. * Verification: If you see an IP address or hostname in that URL that corresponds to one of your CN or BE pods (usually on port 8040), it proves that the data was processed by that node, not the FE. The FE only coordinates; it never generates that ErrorURL logic for the data itself. 2. Inspect the 
fe-proxy
Logs You can watch the proxy move the request from the FE to a CN in real-time. Find your proxy pod and tail the logs:
Copy code
bash
kubectl logs -f <fe-proxy-pod-name>
When you run a load, you should see two entries for a single 
curl
command: 1. A 
PUT
request to the FE service. 2. An immediate subsequent request to a CN IP on port 8040. If you see the second request going to a 10.x.x.x IP (internal Pod IP) on port 8040, the proxy is successfully following the redirect. 3. The "Force Redirect" Test If you want to see the redirect with your own eyes, you have to bypass the proxy. You can do this by running a 
curl
from inside a pod that is in the same Kubernetes namespace (like a temporary sleep pod):
Copy code
bash
# Run this from a pod inside the cluster
curl -v --location-trusted -u 'user:pass' \
    -T /tmp/test.csv \
    -XPUT <http://kube-starrocks-fe-service:8030/api/db/table/_stream_load>
Because you are talking directly to the 
fe-service
on port 
8030
(bypassing the proxy on 
8080
), you will see the 
307 Temporary Redirect
in the verbose output. Summary for your Heap Issue If your Stream Loads are finishing successfully through the 
fe-proxy
, the "Data Proxying" theory is officially debunked. The data is moving through Nginx, not Java. Your 87% Heap usage is strictly a Metadata/Label problem. The FE is keeping a record of every load "Label" you've sent. If you do 100 loads a minute, that’s 144,000 strings stored in the JVM every day. * Immediate Fix: 
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
* Long-term Fix: Increase FE memory or decrease the frequency of small loads (batch more data per 
curl
command). References • loading/StreamLoad.md • quick_start/helm.md • administration/management/FE_configuration.md • faq/shared_data_faq.md References • loading/StreamLoad.md • quick_start/helm.md • administration/management/FE_configuration.md • faq/shared_data_faq.md ) References • quick_start/helm.md • loading/StreamLoad.md • administration/management/FE_configuration.md • administration/management/resource_management/Load_balance.md
g
@Rocky i do not see 8040 anywhere in my fe-proxy logs
r
The reason you do not see 8040 (the default BE/CN HTTP port) in your 
fe-proxy
logs is a strong indicator of why your FE Heap is so high. There are two likely scenarios for why 8040 is missing from your proxy logs: Scenario A: The FE is proxying the data (Most Likely) In this scenario, the redirect (307) never happens because the FE is configured to handle the entire data stream itself. The flow looks like this: 
Client
$\rightarrow$ 
fe-proxy
(Nginx) $\rightarrow$ 
FE
(Java) $\rightarrow$ 
CN/BE
. In this case, the FE acts as a middleman for every byte of data. This data is buffered in the FE's JVM Heap, which perfectly explains why your heap usage is at 87%. How to verify: Run this SQL command in your MySQL client:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE "proxy_stream_load";
* If it is `true`: The FE is proxying all Stream Load data. This is why you don't see 8040 in the logs (the proxy only talks to the FE) and why your heap is high. * Recommendation: Set it to 
false
using 
ADMIN SET FRONTEND CONFIG ("proxy_stream_load" = "false");
. Scenario B: The 
fe-proxy
is not following redirects If 
proxy_stream_load
is already 
false
, then the FE is sending a 307 redirect, but your 
fe-proxy
(Nginx) might not be configured to "follow" it internally. In some older or custom Helm configurations, if Nginx doesn't handle the 307, it passes the 307 back to the client. If your client (curl) then successfully loads the data, it means your client somehow reached the BE/CN directly. However, if your client is outside the cluster and doesn't have a route to the internal Pod IPs, the load would simply fail. Why does this matter for your Heap? If you are seeing successful loads but no 8040 in the 
fe-proxy
logs: 1. If the FE is proxying (Scenario A): The data is filling your Java Heap. You should turn off 
proxy_stream_load
. 2. If the FE is NOT proxying (Scenario B): The 87% heap is purely metadata (labels, transaction records, and tablet stats). In a shared-data cluster, the FE Leader maintains a lot of state in memory to coordinate with object storage. What to check next: 1. Check 
proxy_stream_load
as shown above. 2. Run 
SHOW PROC '/statistic';
and look at the number of 
UnfinishedBackends
or the sheer number of tables/tablets. 3. Check the 
fe-proxy
pod's configuration to see if it has the 
error_page 307 = @handle_redirect;
logic. If it doesn't, it is just a simple pass-through. Summary: If 
proxy_stream_load
is 
true
, your heap issue is the data itself. If it's 
false
, your heap issue is the number of "Labels" (metadata) being kept in memory from frequent small loads. References • administration/management/FE_configuration.md • loading/StreamLoad.md • quick_start/helm.md • faq/shared_data_faq.md ) References • administration/management/FE_configuration.md • loading/StreamLoad.md • quick_start/helm.md • faq/shared_data_faq.md ) References • administration/management/FE_configuration.md • quick_start/helm.md
g
@Rocky i don't see a value for proxy_stream_load
r
If you do not see 8040 (the BE/CN HTTP port) in your 
fe-proxy
logs, it means the Nginx proxy is not successfully intercepting and following the redirect to the compute nodes. I should clarify: I previously mentioned a configuration named 
proxy_stream_load
. Upon verifying the documentation, that specific configuration does not exist in StarRocks. I apologize for the confusion. In StarRocks, the FE (Java) does not natively proxy Stream Load data. It strictly acts as a coordinator that returns a 
307 Temporary Redirect
pointing to a BE or CN. If a load is succeeding through your 
fe-proxy
but you don't see the 8040 port in the logs, here is what is likely happening and how to verify it: 1. Verify the actual Data Node Check the JSON response of a successful Stream Load. Look for the 
ErrorURL
 field: * Example: 
"ErrorURL": "<http://10.x.x.y:8040/api/_load_error_log?...>"
* If the IP/hostname in that URL is a CN or BE pod, then the data definitely went to that node. If the data is reaching the CN, it means the redirect is working, but your Nginx logging configuration might only be showing the initial client request to the FE (port 8030) and not the internal sub-request to the CN (port 8040). 2. Check the Nginx Log Format The 
fe-proxy
is an Nginx container. By default, it might only log the request received from the client. To see where Nginx is sending the data, you would need to look at the upstream logs. * If you see a log entry with a 
307
status code, it means Nginx passed the redirect back to your client. * If your client followed it and the load succeeded, it means your client has a direct network path to the CN pods. 3. Why the FE Heap is still 87% Since the FE does not proxy data bytes, your high heap usage is metadata-related. In a shared-data cluster, the FE Leader is the "brain" for every transaction and object storage interaction. If you are performing many small Stream Loads, the FE Heap fills up with: * Load Labels: StarRocks keeps a record of every load label to prevent duplicates. * Transaction State: Metadata for every completed or failed load. * Tablet/Version Metadata: Tracking data in S3/Object Storage. What you should do to lower the Heap: 1. Reduce Label Retention: By default, StarRocks keeps labels for 3 days. If you have high-frequency loads, reduce this to 1 hour:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
2. Check Task History: See how many tasks are being kept in memory:
Copy code
sql
    SHOW PROC '/statistic';
If the number of tables or tablets is very high, you may simply need to increase the FE JVM Heap size (e.g., 
-Xmx16g
or higher) to accommodate the metadata. Summary If you don't see 8040 in the proxy logs but the load succeeds, your data is successfully reaching the CNs. Your FE memory issue is almost certainly caused by metadata/label retention, not the data transfer itself. References • administration/management/FE_configuration.md • loading/StreamLoad.md • quick_start/helm.md • faq/shared_data_faq.md References • loading/loading_introduction/loading_considerations.md • administration/management/FE_configuration.md • quick_start/helm.md • administration/management/resource_management/Load_balance.md
g
@Rocky instead of using the fe-proxy, should i just add the stream load port to my fe configs like
Copy code
http_port = 8030                                                                                                                            
rpc_port = 9020                                                                                                                             
query_port = 9030                                                                                                                           
edit_log_port = 9010
r
No, you should not replace the 
fe-proxy
with those direct port configurations if you are loading data from outside your Kubernetes cluster. Those ports (
8030
, 
9020
, etc.) are the internal ports that the StarRocks FE process uses to listen for traffic. They are likely already in your configuration. However, simply exposing them via a LoadBalancer will not fix the "307 Redirect" issue for external clients. Why you still need the 
fe-proxy
 When you send a Stream Load to the FE (port 8030), the FE does not process the data. Instead: 1. FE returns a 
307 Temporary Redirect
 containing the internal Pod IP or hostname of a CN/BE (e.g., `http://10.42.x.x:8040/...`). 2. If you are outside the cluster, your 
curl
command (the client) will try to follow that redirect but fail because your laptop/server cannot reach that internal 
10.x.x.x
IP. The 
fe-proxy
(Nginx) is specifically designed to solve this. It sits inside the cluster and handles the "jump" to the CN for you, or rewrites the redirect so that the data transfer happens entirely within the K8s network. Why you don't see 
8040
in the logs If your loads are succeeding but you don't see 
8040
in the 
fe-proxy
logs, it is likely due to one of the following: * Logging Level: The default Nginx configuration in the 
fe-proxy
may only log the initial 
PUT
request from the client and not the "upstream" connection to the CN. * Proxy Logic: In some versions of the Helm chart, the proxy is configured to rewrite the header. Check your 
curl
response for the 
ErrorURL
. If that URL shows a CN IP, then the data definitely bypassed the FE. How to verify once and for all Run this command from your external client:
Copy code
bash
curl -v --location-trusted -u 'root:password' \
    -T your_file.csv \
    -H "label:test_$(date +%s)" \
    http://<FE_PROXY_LB_IP>:8080/api/db/table/_stream_load
Look for these in the output: 1. Status 307: You should see the proxy receive a 307 from the FE. 2. The "Location" header: See what IP/port the FE is suggesting. 3. JSON Result: If the 
ErrorURL
in the final JSON response contains port 
8040
, the data was processed by a CN/BE. About your FE Heap (87%) Since the FE cannot be configured to proxy data bytes (it only ever redirects), your high memory usage is not caused by the data stream. It is caused by Metadata: * Labels: Every 
label
you send is stored in memory. If you load frequently, reduce the retention: 
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
* JVM Overhead: If you have many tables/tablets, the FE needs a larger Heap. Ensure your 
FE_JVM_OPTS
is set to at least 
-Xmx8g
or 
-Xmx16g
. Summary: Keep the 
fe-proxy
for connectivity. Solve the Heap issue by cleaning up load labels and increasing the FE memory limit in your Helm. References • quick_start/helm.md • loading/StreamLoad.md • administration/management/FE_configuration.md • deployment/sr_operator.md ) References • administration/management/resource_management/Load_balance.md • deployment/sr_operator.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md
g
@Rocky i'm using an azure managed LB, could this be the problem? this is in my helm service.beta.kubernetes.io/azure-load-balancer-internal: "true"
r
The Azure Managed Internal Load Balancer and your Helm configuration explain exactly why you are seeing these two symptoms (missing 8040 logs and high FE heap). 1. Why you don't see "8040" in logs If you applied the 
<http://service.beta.kubernetes.io/azure-load-balancer-internal|service.beta.kubernetes.io/azure-load-balancer-internal>: "true"
annotation to your FE service (port 8030) rather than the fe-proxy service (port 8080), here is what is happening: * The Flow: Your client hits the Azure LB IP $\rightarrow$ FE (Port 8030). * The Redirect: The FE returns a 
307 Temporary Redirect
containing the internal Pod IP of a BE/CN (e.g., `http://10.x.x.x:8040`). * The Bypassing: Because your client is inside the Azure VNet (or connected via VPN/ExpressRoute), it can reach that 
10.x.x.x
IP directly. * The Result: The client follows the redirect and sends the data straight to the BE. The 
fe-proxy
(Nginx) is never touched for the second half of the request, so 8040 never appears in any proxy logs. How to fix connectivity: If your loads are already succeeding, your network is fine. If you want the proxy to handle everything (to keep traffic centralized), you must point your client to the fe-proxy service IP (port 8080) instead of the FE service IP. 2. Why the FE Heap is at 87% Since the FE does not proxy data bytes (it only sends the 307 redirect), your heap is NOT high because of the data transfer. It is high because of Metadata Bloat. In a shared-data cluster on Azure (using Blob Storage), the FE Leader must track every single "Label" and "Transaction" to ensure data consistency. If you are doing frequent Stream Loads, your FE memory is being filled by: 1. Load Labels: StarRocks stores every label for 3 days by default. 2. Tablet Reports: In large clusters, the FE keeps massive maps of tablet locations in memory. 3. Transaction Metadata: Metadata for every version of every tablet created by your loads. 3. Recommended Actions Do NOT change your ports (like adding 8030/9020 to the config). Instead, address the metadata and memory limits: 1. Reduce Label Retention: This is the most common fix for high FE heap during Stream Loads. Shorten the time StarRocks remembers old load labels from 3 days to 1 hour:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
2. Increase FE JVM Heap: In the StarRocks Helm chart, ensure your 
FE_JVM_OPTS
provides enough room for metadata. For a production cluster, you should aim for at least 16GB (
-Xmx16g
). 3. Check for "Metadata Ghosting": Run the following to see what is eating memory:
Copy code
sql
    -- Check how many labels are currently in the system
    SHOW PROC '/statistic';
4. Use MemoryUsageTracker (v3.3.7+): If you are on a recent version, check your 
fe/log/fe.log
for 
MemoryUsageTracker
entries. It will explicitly tell you which module (e.g., 
StreamLoadMgr
or 
TabletManager
) is consuming the most heap. Summary: Your Azure LB setup is allowing your client to bypass the proxy for the actual data transfer (which is why 8040 is missing from logs). Your FE Heap issue is a separate problem caused by the high volume of load metadata stored in Java memory. References • quick_start/helm.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md • loading/StreamLoad.md • deployment/sr_operator.md ) References • quick_start/helm.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md • loading/StreamLoad.md
g
@Rocky it seems if it wasn't redirecting, my heap issues would be more severe?
r
Yes, your intuition is correct. If the FE were actually proxying the raw data bytes of every Stream Load, your heap issues would be significantly worse and you would likely be seeing frequent Out Of Memory (OOM) crashes. Here is why your "redirecting" setup is actually protecting the FE, and why you still have a heap issue: 1. Data vs. Metadata * If FE Proxied Data (Severe): The FE process (Java) would have to buffer incoming data chunks, manage thousands of active HTTP streams, and serialize/deserialize those bytes before forwarding them to the CN. This would consume massive amounts of heap memory and CPU for every concurrent load. * The Current Redirect (Normal): The FE only spends a few milliseconds on your request. It checks permissions, generates a transaction ID, and sends a tiny "307 Redirect" packet back to the client. The actual gigabytes of data never enter the FE's memory. 2. Why your Heap is still at 87% Even though the data isn't flowing through the FE, the metadata from your loads is staying behind in the heap. If you are doing frequent loads, the FE keeps track of: * 
StreamLoadTask
objects: Every load you perform creates an entry in the FE's memory. By default, StarRocks keeps thousands of these. * `stream_load_task_keep_max_num`: This defaults to 1000 tasks. If you have many loads, this object pool takes up permanent heap space. * `stream_load_task_keep_max_second`: This defaults to 3 days (259,200 seconds). The FE remembers every load label for 3 days to prevent duplicates. 3. How to "Deflate" the Heap Since you are likely loading frequently, you should force the FE to clean up this metadata much faster. Run these commands to see immediate relief:
Copy code
sql
-- Keep only 1 hour of load history in memory instead of 3 days
ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_second" = "3600");

-- Keep only 500 recent tasks in memory instead of 1000
ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_num" = "500");

-- Reduce general label retention (used for all load types)
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
4. Verify what is using the memory If the heap stays high after these changes, you can verify exactly which module is the "hoarder" by looking at the MemoryUsageTracker (if you are on v3.3.7+). Check your 
fe/log/fe.log
for lines like: 
MemoryUsageTracker ... StreamLoadMgr: XXX MB, TabletManager: YYY MB
Summary: You are correct—the 307 redirect is the only reason your FE is still standing. Your current heap issue is caused by the FE "remembering" too many old load tasks and labels. Lowering the retention settings above is the standard fix. References • administration/management/FE_configuration.md • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • loading/StreamLoad.md References • faq/Others.md • knowledge/trouble_shooting/fe_memory_problems.md • loading/loading_introduction/loading_considerations.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md
g
@Rocky if stream load making heap use go up, why is it better than jdbc?
r
Stream Load is significantly better than JDBC (
INSERT INTO VALUES
) for bulk ingestion because of how they split work between the Frontend (FE) and Backend (BE). While you see the FE heap increasing with Stream Load, using JDBC for the same volume of data would likely be much worse. Here is why Stream Load is the preferred path: 1. SQL Parsing Overhead (The "JDBC Trap") When you use JDBC to run `INSERT INTO table VALUES (...), (...);`: * FE Bottleneck: The FE must parse the entire SQL string. If your JDBC batch has 10,000 rows, the FE has to tokenize and parse a massive text string. This is extremely CPU and Memory intensive for the Java process. * Data in Memory: The FE must hold the parsed row data in memory while it generates a query plan and determines which BEs should receive which rows. * Serial Processing: JDBC inserts are often processed row-by-row or in smaller batches, which leads to massive transaction overhead and "too many versions" errors in StarRocks. 2. The Stream Load "Shortcut" Stream Load is a specialized high-performance pipe: * No SQL Parsing: The FE does not parse your data. It only parses the HTTP headers (table name, column mapping, etc.). * Direct Data Transfer: As discussed, the FE just gives the client a "Redirect." The actual gigabytes of CSV/JSON data go directly from your client to the BE. The FE never even sees the data bytes. * Metadata Only: The heap usage you see is just the "receipt" (the Label, the Transaction ID, and the status). This is much "lighter" than the memory used to parse SQL. 3. Comparison at Scale Feature | JDBC (
INSERT INTO
) | Stream Load FE Memory Usage | High (Parses every row/value) | Low (Stores metadata/labels only) FE CPU Usage | High (SQL Tokenizing/Parsing) | Negligible (Header parsing) Data Path | Client $\rightarrow$ FE $\rightarrow$ BE | Client $\rightarrow$ BE (after 307 redirect) Throughput | Low (KB/s to MB/s) | High (GBs/s across cluster) Transaction Cost | High (One txn per INSERT) | Low (Optimized for bulk) Why your heap is high even with Stream Load The reason your heap is at 87% isn't because Stream Load is "heavy," but because of Retention. * If you do 1,000 Stream Loads per hour, and StarRocks keeps the record of those 1,000 loads in memory for 3 days (the default), you eventually have 72,000 "receipts" sitting in the Java Heap. * With JDBC, you would likely be hitting OOM (Out of Memory) or CPU spikes long before you reached that volume. Summary Recommendation Stick with Stream Load. It is the most efficient way to get data into StarRocks. To fix the heap issue, simply tell the FE to forget about old loads faster:
Copy code
sql
-- Immediate relief: reduce retention from 3 days to 1 hour
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "3600");
ADMIN SET FRONTEND CONFIG ("stream_load_task_keep_max_second" = "3600");
References • loading/loading_introduction/loading_considerations.md • loading/StreamLoad.md • unloading/arrow_flight.md • administration/management/FE_configuration.md References • loading/StreamLoad.md • unloading/arrow_flight.md • loading/Stream_Load_transaction_interface.md • loading/loading_introduction/loading_concepts.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md
g
@Rocky looking at grafana, I see my leader FE JVM young commited hanging around 10g then then it will gradually drop below 8g then shoot back up to 10g then hours later start repeating the process, is this normal behavior?
r
Yes, this is completely normal behavior for a Java-based process like the StarRocks Frontend (FE), especially when using the modern G1 Garbage Collector (G1GC). What you are seeing is the JVM dynamically tuning its internal memory pools to balance performance and efficiency. Here is exactly why the graph looks that way: 1. "Committed" vs. "Used" It is important to distinguish between these two metrics: * Used: The actual bytes of memory occupied by live objects (your metadata, labels, etc.). * Committed: The amount of memory the JVM has "reserved" from the operating system. Think of it as the current "budget" allocated to a specific pool (like the Young Generation). 2. Why the 8GB to 10GB cycle? StarRocks FEs are usually configured with G1GC. Unlike older collectors that use fixed-size pools, G1GC treats the heap as a collection of equal-sized regions and dynamically classifies them as "Young" or "Old" on the fly. * The "Shoot back up" (10GB): As you perform Stream Loads or queries, many short-lived objects are created (e.g., HTTP headers, plan fragments). The JVM expands the Young Generation's committed size to handle the burst without triggering frequent, aggressive collections. * The "Gradual drop" (8GB): After a period of stability or after a successful "Mixed Collection," the JVM realizes it doesn't need all 10GB reserved for the Young Gen. To be efficient, it releases some of that committed memory back to the general heap pool or the OS. 3. Why it repeats every few hours This usually correlates with your background maintenance tasks in StarRocks: * Label Cleanup: By default, StarRocks runs a cleanup thread every few hours (controlled by 
label_clean_interval_second
) to purge old load metadata. * Checkpointing: Every 50,000 journal entries (or roughly every hour depending on load), the FE Leader performs a Checkpoint to consolidate its image. This creates a temporary spike in object allocation followed by a significant cleanup. When should you worry? You only need to take action if: 1. The "Used" memory (not committed) stays at 80-90% and never drops after a collection. 2. The "Old Generation" (where long-term metadata lives) is steadily climbing without ever dropping. 3. FE "Stop-the-World" (STW) pauses exceed 1 second (visible in the "GC Time" or "GC Duration" panels in Grafana). Summary: Your graph shows a healthy JVM "breathing." The 10GB $\leftrightarrow$ 8GB fluctuation is the JVM successfully resizing itself to accommodate your workload. If your "Used" heap is still around 87% as mentioned earlier, focus on the 
label_keep_max_second
setting to reduce the baseline "Used" memory, which will give the "Young" generation more room to breathe. References • administration/management/monitoring/Monitor_and_Alert.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md References • administration/management/FE_configuration.md • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md
2 Views
Open in Slack
PreviousNext
Powered by