http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I have K8S based setup with 3 FE an...
# questions-and-troubleshooting
s
@Rocky I have K8S based setup with 3 FE and 3 BE pod and during FE/BE sequential restarts (forced by changing pod annotations), my ingestion service failed to push data using HTTP Stream Load API. The tables it writes to have 
replication_num=2
Is this a correct setup for HA ? The DDL is
Copy code
CREATE TABLE `leg_staging` (
  `account_key` bigint(20) NOT NULL COMMENT "",
  `leg_id` varchar(36) NOT NULL COMMENT "",
  `ts` datetime NOT NULL COMMENT "",
  `parent_id` varchar(36) NULL COMMENT "",
  `user_key` bigint(20) NULL COMMENT "",
  `initiator` boolean NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`account_key`, `leg_id`, `ts`)
PARTITION BY time_slice(ts, 24, 'hour', 'floor')
DISTRIBUTED BY HASH(`account_key`) BUCKETS 17
PROPERTIES (
"colocate_with" = "account_key_colocation_group",
"compression" = "LZ4",
"fast_schema_evolution" = "true",
"partition_ttl" = "15 day",
"replicated_storage" = "true",
"replication_num" = "2"
);
r
Hello! I'm Rocky. Based on your setup and the DDL provided, your ingestion service is failing during rolling restarts because of the Write Quorum requirement. 1. Why it's failing In StarRocks, the default 
write_quorum
for any table is 
MAJORITY
. The majority is calculated as 
(replication_num / 2) + 1
. * For 
replication_num = 2
, the majority is 2. * This means both replicas must be successfully written for the Stream Load to return a success. When you perform a sequential restart (rolling update) in a 3-BE cluster: 1. One BE pod goes down. 2. Any tablet that had one of its two replicas on that BE now only has 1 replica available in the cluster. 3. Since 1 is less than the required majority (2), the write operation fails. 2. Is this a correct setup for HA? No. A 
replication_num = 2
setup is not sufficient for High Availability (HA) during maintenance or node failures. For a production cluster with 3 BEs, you should aim for a setup where the service remains available even if one node is down. 3. Recommendations Option A: Increase Replication (Recommended) Change your 
replication_num
to 3. In a 3-BE cluster with 3 replicas: * The majority is 2. * If 1 BE is down, 2 replicas are still available, satisfying the quorum. * Your ingestion will continue to work during rolling restarts. You can alter your existing table:
Copy code
sql
ALTER TABLE leg_staging SET ("replication_num" = "3");
Option B: Change Write Quorum (Lower Reliability) If you must keep 2 replicas (e.g., due to storage constraints), you can change the 
write_quorum
property to 
ONE
. * This allows the write to succeed as long as at least one replica is written. * Warning: This reduces data reliability. If the single node that received the data fails before it can sync to the other replica (once it comes back online), you could lose that batch of data.
Copy code
sql
ALTER TABLE leg_staging SET ("write_quorum" = "ONE");
4. Other HA Considerations * FE HA: Your 3 FE setup is correct for HA. During a rolling restart, 2 FEs remain active, which maintains the FE quorum (majority of 3 is 2). * Load Balancing: Ensure your ingestion service is hitting a Load Balancer (like a Kubernetes Service or an HAProxy) that points to the FEs. If you are hardcoding a single FE IP, the service will fail when that specific FE restarts. * replicated_storage: You have this set to 
true
(Single Leader Replication). This is good for performance (reduces BE CPU usage during load), but it does not change the quorum requirements mentioned above. Summary: To fix the ingestion failures during restarts without sacrificing data safety, the best path is to set 
replication_num = 3
. References • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md • administration/Meta_recovery.md • faq/Others.md • administration/management/BE_configuration.md • knowledge/usage/best-practice.md
s
@Rocky If I keep replication_num=2 and had instead 4 BE, would it work ?
r
No, simply increasing the number of BE nodes to 4 while keeping 
replication_num=2
will not solve the problem during rolling restarts. Here is why: 1. The Math of Majority Quorum As mentioned, the default 
write_quorum
is 
MAJORITY
. The formula for a majority is 
(replication_num / 2) + 1
. * For 
replication_num = 2
, the majority is 2. * This means every single replica must be online and available for the write to succeed. Effectively, for 2 replicas, 
MAJORITY
behaves the same as 
ALL
. 2. The Impact of a Rolling Restart In a cluster with 4 BEs and a table with 2 replicas: 1. StarRocks distributes the 2 replicas of each tablet across 2 different BEs (out of the 4 available). 2. When you restart 1 BE pod, any tablet that had one of its replicas on that specific pod now only has 1 replica available in the cluster. 3. Since the required quorum is 2, any ingestion attempt involving those tablets will fail. Because your table data is distributed (sharded) across many tablets, it is almost certain that at least some of your tablets will have a replica on the BE being restarted, causing the entire Stream Load job to fail. 3. Comparison of Setup Options Setup | BE Count | Replication | Write Quorum | Survival during Restart? Current | 3 | 2 | MAJORITY (2) | No Proposed | 4 | 2 | MAJORITY (2) | No Recommended HA | 3 or 4 | 3 | MAJORITY (2) | Yes (1 node can be down) Lower Reliability | 3 or 4 | 2 | ONE (1) | Yes (Data risk if node fails permanently) Conclusion To achieve true High Availability during rolling restarts, you have two real paths: * Increase 
replication_num
to 3: This is the standard production recommendation. It allows 1 BE to be offline while still meeting the majority quorum (2 out of 3). * Change 
write_quorum
to `ONE`: This allows the write to succeed if only 1 replica is up. However, if the node that received the data crashes before it can sync to the second replica, you could lose data. For a K8s environment where pods are frequently rescheduled or restarted, 
replication_num=3
 is the most stable and correct configuration. References • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • administration/management/graceful_exit.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • administration/Meta_recovery.md
s
@Rocky I recreated the tables with 
replication_num=3
and as soon as one of the BE pod was reloading during ingestion it was failing. A retry did not help while it was still loading:
Copy code
2026/04/30 14:53:03 ingestion failed: sending leg_staging batch: stream load to leg_staging failed: Couldn't open transport for starrocks-insights-fe-2.starrocks-insights-fe-search.ucc-core.svc.cluster.local:9020 (socket open() error: Connection refused)
response: {
    "TxnId": -1,
    "Label": "e51ee4f2-0ba7-491f-a4ce-eb824fa52e8b",
    "Db": "test",
    "Table": "leg_staging",
    "Status": "Fail",
    "Message": "Couldn't open transport for starrocks-insights-fe-2.starrocks-insights-fe-search.ucc-core.svc.cluster.local:9020 (socket open() error: Connection refused)",
    "NumberTotalRows": 0,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 0,
    "LoadTimeMs": 0,
    "BeginTxnTimeMs": 0,
    "StreamLoadPlanTimeMs": 0,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 0,
    "CommitAndPublishTimeMs": 0
}
r
It looks like you've successfully addressed the BE quorum issue by moving to 
replication_num=3
. However, you are now hitting a separate High Availability (HA) issue at the FE (Frontend) layer. 1. The Error Analysis The error message provides a crucial hint: 
Couldn't open transport for starrocks-insights-fe-2...:9020 (socket open() error: Connection refused)
* Port 9020: This is the FE Thrift RPC port (
rpc_port
). It is used by many programmatic clients (like the StarRocks Go, Java, or C++ SDKs) to coordinate transactions and fetch metadata. * The Target: Your service is trying to connect specifically to 
starrocks-insights-fe-2
. * The Failure: Because you are doing a sequential restart, when 
fe-2
is being rebooted, it is physically unavailable. Since your ingestion service is targeting that specific pod's DNS name, it gets a "Connection refused." 2. Why your retry failed If your ingestion service is configured with a hardcoded list of FE pod addresses (or just one), it will continue to fail as long as that specific pod is down. A "retry" in your application code only works if it knows how to failover to a different FE (like 
fe-0
or 
fe-1
), which are likely still healthy. 3. How to fix FE HA in Kubernetes To make your ingestion resilient to FE restarts, you should not point your application to individual pod names. Instead, follow these steps: Step A: Use the Kubernetes Service DNS Instead of 
starrocks-insights-fe-2...
, use the DNS name of the Kubernetes Service that sits in front of your FEs. * Incorrect: 
starrocks-insights-fe-2.starrocks-insights-fe-search.ucc-core.svc.cluster.local
* Correct: 
starrocks-insights-fe-service.ucc-core.svc.cluster.local
(Replace with your actual service name). Kubernetes Services automatically track which pods are "Ready." When a pod starts its restart process, it fails its readiness probe and the K8S Service stops sending traffic to it, routing your requests to the remaining healthy FEs instead. Step B: Check Readiness Probes Ensure your FE Deployment/StatefulSet has a proper 
readinessProbe
configured. A typical FE readiness probe checks the HTTP port (8030) or the MySQL port (9030).
Copy code
yaml
readinessProbe:
  tcpSocket:
    port: 9030
  initialDelaySeconds: 5
  periodSeconds: 10
Without a readiness probe, Kubernetes might still send traffic to a pod that is starting up but not yet ready to handle Thrift RPC requests on port 9020. Step C: Connection Strings (if using a client library) If you are using a library that allows multiple FE hosts, provide all three: 
fe-0:9030,fe-1:9030,fe-2:9030
(or the RPC equivalents). Most StarRocks-compatible drivers will automatically try the next host in the list if one is unreachable. Summary Your 
replication_num=3
change was correct and necessary for the BEs to survive the restart. The current failure is because your ingestion service is "pinned" to a specific FE pod that is going offline. Moving to a load-balanced Service address will allow your ingestion to automatically skip the restarting FE and use the ones that are still up. References • deployment/environment_configurations.md • quick_start/helm.md • administration/management/FE_configuration.md • loading/StreamLoad.md • faq/loading/Stream_load_faq.md References • administration/management/FE_configuration.md • deployment/deploy_manually.md • deployment/environment_configurations.md
s
@Rocky The ingestion service was configured on a Gateway API endpoint that forwards Stream Load request to the FE Proxy K8S service 
starrocks-insights-fe-proxy-service
. The FE proxy pods did not reload. I'm using the operator default values for the k8s readiness probes.
r
The error you are seeing confirms that you have moved past the BE replica issue and are now encountering a Frontend (FE) Leader Election bottleneck. 1. The Root Cause: Leader Transition In StarRocks, every transaction (including a Stream Load) must be managed by the Leader FE. * Follower Proxying: When you send a Stream Load to the 
fe-proxy
service, the request is forwarded to a random FE (e.g., 
fe-0
). If 
fe-0
is a Follower, it must communicate with the Leader (e.g., 
fe-2
) via Thrift RPC on port 9020 to begin the transaction. * The Error: Your error message 
Couldn't open transport for ...fe-2...:9020
is a Thrift network error. It means the FE that received your request tried to talk to the Leader (
fe-2
), but 
fe-2
was physically restarting and the connection was refused. 2. Why Retries Failed When an FE Leader is killed during a rolling restart: 1. Detection: The remaining FEs take a few seconds to realize the Leader is gone (based on heartbeats). 2. Election: The remaining FEs (
fe-0
and 
fe-1
) hold an election to choose a new Leader. 3. Availability Gap: During this election window (typically 5 to 15 seconds), the cluster has no Leader and cannot accept any new writes. If your ingestion service retried immediately or very quickly, it likely hit this "no leader" window or still tried to talk to the old leader before the new one was fully operational. 3. How to achieve HA during FE restarts A. Implement an Exponential Backoff Retry Since a Leader election is unavoidable when the Leader pod restarts, your ingestion client must be prepared to wait. * Recommendation: Use a retry logic that waits at least 10-20 seconds upon a "Connection Refused" or "Master not ready" error. This gives the FE cluster enough time to stabilize the new Leader. B. Validate "Sequential" Restart Timing When you force a restart by changing annotations, Kubernetes might move to the next pod as soon as the previous pod is technically "Running." However, a StarRocks FE is not truly "Ready" until it has synchronized metadata. * Check: Ensure your rolling update waits for the FE to be 
Ready
in Kubernetes. The StarRocks Operator's default readiness probe usually checks the MySQL port (9030). * Wait for Health: Before restarting the next FE pod, you should verify the cluster status:
Copy code
sql
    SHOW FRONTENDS;
Ensure all FEs are 
Alive: true
and one is clearly marked as 
IsMaster: true
before proceeding to the next pod. C. BE Graceful Decommission Since you also mentioned BE restarts: Even with 
replication_num=3
, if you kill a BE abruptly, active Stream Loads may fail. * Best Practice: In K8s, use the Operator's ability to manage restarts. If you are doing this manually, ensure you have a 
preStop
hook or that the Operator is configured to handle "Graceful Shutdown." StarRocks BEs need time to finish active fragments and migrate primary tablet roles. Summary Your setup (
replication_num=3
and 3 FEs) is technically correct for HA. The failures you are seeing are "transient" errors caused by the physical time required for the FE cluster to elect a new Leader. 1. Replication 3 fixed the data availability (BE) problem. 2. Retry logic with backoff (10s+) will solve the FE election problem. 3. Slower rolling updates (checking 
SHOW FRONTENDS
between pods) will ensure you don't accidentally take down the majority of the FEs at once. References • administration/management/FE_configuration.md • administration/Meta_recovery.md • loading/StreamLoad.md • faq/loading/Stream_load_faq.md • deployment/deploy_manually.md References
s
@Rocky On FE leader election process, do read query (specifically over HTTP, not MySQL) still work ?
r
The short answer is no. During the window of a Leader election, read queries—including those over HTTP (like the HTTP SQL API)—will generally fail or time out. Here is a detailed breakdown of why this happens and how it affects your HTTP traffic: 1. Metadata Consistency (The Default Behavior) StarRocks FEs use a distributed metadata store (BDB JE). By default, Follower FEs follow a strict consistency rule: they will not serve any requests (read or write) if they cannot verify that their local metadata is up-to-date. * During a leader election, there is no active Leader to "heartbeat" against. * Followers enter a transition state where they are unsure if they have the latest "journaled" metadata. * In this state, the FE typically rejects queries with errors like 
Master not ready
or 
UnknownMasterException
. 2. HTTP Interface vs. MySQL Interface There is no difference in availability between the MySQL protocol and the HTTP interface during this window. Both rely on the same internal FE "Catalog" and metadata layer. If the metadata layer is in an "Election" state, the HTTP server will either: * Return a 
500 Internal Server Error
with a message about the master not being available. * Hang until the connection times out if the election takes longer than the client's timeout. 3. The Role of the FE-Proxy Since you are using the `starrocks-insights-fe-proxy-service`: * The proxy sits in front of all 3 FEs. * When the Leader (
fe-2
) goes down, the proxy might still forward a request to 
fe-0
or 
fe-1
. * However, 
fe-0
and 
fe-1
will be busy performing the election. They will refuse the request forwarded by the proxy until one of them successfully becomes the new Leader. 4. Can you make reads work during election? There is a configuration parameter called 
ignore_meta_check
(default: 
false
). * If set to 
true
on the Follower FEs, they will continue to serve read queries even if they lose connection to the Leader or during an election. * The Risk: This allows "Stale Reads." You might query data that was committed to the Leader just milliseconds before it crashed, but which hasn't yet reached the Follower you are querying. * The Benefit: It provides higher availability for read-only workloads during rolling restarts. Summary of Availability during FE Leader Election Request Type | Default Behavior | With 
ignore_meta_check=true
 Stream Load (Write) | Fails (Requires Leader) | Fails (Still requires Leader) MySQL Query (Read) | Fails (During election window) | Works (Returns potentially stale data) HTTP SQL API (Read) | Fails (During election window) | Works (Returns potentially stale data) FE Health Check | Works (Usually returns status) | Works Recommendation: For the best HA experience, do not rely on 
ignore_meta_check
unless your application can tolerate slightly stale data. Instead, ensure your application or your Gateway API has a retry policy with an exponential backoff (e.g., retry after 5s, 10s) to bridge the 5–15 second gap required for a new Leader to be elected. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • sql-reference/http_sql_api.md References • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • deployment/deploy_manually.md
s
@Rocky what are the concrete drawbacks of 
ignore_meta_check=true
it seems like getting somewhat stale data for reporting is okay to me
r
Setting 
ignore_meta_check=true
is often tempting to maintain uptime during rolling restarts, but it carries several "hidden" technical risks beyond just seeing slightly old data. While "stale data" might be acceptable for a report, "metadata staleness" can cause queries to fail entirely. Here are the concrete drawbacks: 1. Hard Query Failures (Version Errors) This is the most significant technical drawback. * How it works: When a query is planned, the FE looks at its local metadata to determine which "version" of the data to ask the BEs for. * The Problem: If the FE is stale, it might ask for 
Version 10
. However, if the Leader and BEs have already moved to 
Version 50
and performed compaction, 
Version 10
might no longer exist on the BEs. * Result: Instead of getting "stale data," your reporting tool receives a hard error like: 
Error: version already been compacted
or 
Error: tablet does not have version
. 2. Invisibility of New Objects Metadata isn't just row data; it's the definition of the cluster itself. If you hit a stale FE: * Missing Partitions: If a scheduled job or dynamic partitioning created a new partition (e.g., for 
2026-04-30
), a stale FE won't know it exists. Your report for "Today" will return 0 results or "Partition not found," even though the data is physically in the cluster. * Missing Tables: If you just created a new reporting table or view, the stale FE will return 
Table 'xyz' does not exist
. 3. Schema Mismatch & Serialization Errors If you performed a schema change (e.g., 
ALTER TABLE ... ADD COLUMN
) while the FE was disconnected: * The stale FE will generate a query plan based on the old schema. * The BEs, which are up-to-date, will try to execute the plan against the new schema. * This often leads to internal serialization errors or "column index out of bounds" errors, causing the query to crash. 4. Permission & Security Gaps Permissions (Grants/Revokes) are part of the metadata. * If you recently revoked a user's access for security reasons, that user could still run queries if they happen to connect to the stale FE. * Conversely, a newly hired analyst might get "Access Denied" errors because the stale FE doesn't know they've been granted permissions yet. 5. Inconsistent Results (The "Flip-Flop" Effect) Since you are using a Load Balancer/Proxy, your reporting tool might hit 
FE-1
(Fresh), then 
FE-0
(Stale). * User Experience: An analyst refreshes their dashboard and sees 1,000,000 rows. They refresh again 10 seconds later and see 950,000 rows because they hit the stale FE. This "disappearing data" behavior often causes more support tickets than a brief 10-second outage. Recommendation Instead of 
ignore_meta_check=true
, it is almost always better to: 1. Increase `meta_delay_toleration_second`: This allows a small window of staleness (e.g., 30–60 seconds) rather than ignoring the check entirely. 2. Use Client-Side Retries: Ensure your reporting tool or Gateway has a retry policy that waits ~10 seconds. This covers the "Election Window" without risking the structural errors mentioned above. References • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md
s
@Rocky even with replication_num=3 a single BE restarting on K8S fails ingestion. Does this have to do with K8S probes not reacting fast enough or timeouts ?
r
The fact that ingestion fails during a single BE restart despite having 
replication_num=3
is a common point of confusion. Triple replication ensures data durability and read availability, but it does not automatically guarantee write transaction continuity if the underlying network connections are severed. The failure usually stems from three main factors: 1. The "Coordinator BE" Bottleneck When you send a Stream Load request to a Frontend (FE), the FE selects one BE to act as the Coordinator. This BE is responsible for: * Receiving the entire data stream from your client. * Parsing and partitioning the data. * Distributing the data to the other BEs (the replicas). If the BE being restarted happens to be the Coordinator for your active load, the transaction will fail immediately because the TCP connection between your client and that BE is broken. StarRocks does not currently support "transparent failover" of an in-flight Stream Load to a different coordinator. 2. Stream Load is a "Synchronous" Transaction By default, StarRocks uses a 
MAJORITY
write quorum. For 
replication_num=3
, this means 2 out of 3 replicas must succeed. However: * Connection Persistence: If a BE is part of a write and it restarts during the data transmission phase, the internal RPC channel used for that load is closed. * Atomicity: Unlike a distributed file system that might silently retry a block, a database transaction is atomic. If any node participating in the "Plan" of that load disappears unexpectedly, the FE or Coordinator BE will often fail the entire job to ensure data integrity rather than trying to re-route mid-stream. 3. K8S Probes vs. FE Heartbeats (The "Blind Spot") There is a timing gap between when Kubernetes knows a pod is "Unready" and when the StarRocks FE knows a BE is "Dead." * K8S Readiness Probe: When a pod receives a 
SIGTERM
(restart), it may stop passing its readiness probe. This removes the pod from the K8S Service (the 
fe-proxy
), but it does not tell the StarRocks FE to stop using that BE for internal tasks. * FE Heartbeat: The FE only marks a BE as 
DEAD
if it misses heartbeats (default every 5 seconds). * The Gap: For several seconds, the FE still thinks the "restarting" BE is healthy and may continue to assign it as a Coordinator or a replica for new Stream Loads. When those loads attempt to connect to the "exiting" BE, they fail. Solutions and Best Practices 1. Upgrade to v3.4+ (Graceful Exit): StarRocks 3.4 introduced an enhanced Graceful Exit mechanism. When a BE receives a 
SIGTERM
, it enters a 
SHUTDOWN
state. It will: * Finish active "fragments" (including loads). * Proactively tell the FE "I am exiting" via heartbeat. _ Reject _new* tasks while finishing old ones. _ _Requirement:* Ensure your StarRocks Operator is configured to allow enough 
terminationGracePeriodSeconds
(e.g., 60s or 120s) in the Pod spec. 2. Use Flink or StarRocks Connectors: If you use the official Flink or Kafka connectors, they have built-in retry logic. If a BE restart kills a connection, the connector will automatically catch the error and retry the mini-batch against the remaining BEs. 3. Implement Client-Side Retries: For raw Stream Load (via 
curl
or custom code), you must implement a retry loop. A 503 or "Connection Refused" during a rolling upgrade should trigger a retry after a 5–10 second delay. 4. Check `write_quorum`: Ensure your table isn't accidentally set to 
write_quorum = ALL
. If it is, the loss of even one BE will 100% guarantee a write failure. You can check this with 
SHOW CREATE TABLE <table_name>;
. Summary Replication ($R=3$) protects your data, but the Stream Load process is a network-sensitive transaction. To achieve zero-downtime ingestion during BE restarts, you must rely on Graceful Exit (v3.4+) or Client-Side Retries to handle the transient connection drops. References
s
@Rocky So SIGTERM sent by K8S do not trigger graceful exits ?
r
Technically, a 
SIGTERM
is the signal that StarRocks uses to trigger a Graceful Exit for the Backend (BE). However, whether Kubernetes successfully triggers a graceful exit depends on your StarRocks version and how your Pod is configured. If your ingestion is failing immediately during a restart, it is likely due to one of the following reasons: 1. Version Requirement (v3.4+) Graceful Exit is only "fully baked" and enabled by default from v3.4 onwards. * Before v3.3: 
SIGTERM
was often handled like a hard stop. * v3.3: Introduced the feature, but it was "flawed" (the Frontend often marked the BE as 
DEAD
too early, killing the load anyway). * v3.4+: The Frontend (FE) now recognizes the 
SHUTDOWN
state and keeps the transaction alive while the BE finishes its work. 2. The PID 1 Problem Kubernetes sends the 
SIGTERM
to PID 1 inside the container. * If your Docker entrypoint is a shell script (e.g., 
ENTRYPOINT ["/bin/sh", "start_be.sh"]
), the shell script becomes PID 1. * Most shells do not propagate signals to their child processes. The BE process never receives the 
SIGTERM
and simply waits until Kubernetes loses patience and sends a 
SIGKILL
30 seconds later. * Fix: Ensure you use a 
preStop
hook in your K8S manifest to explicitly call the stop script:
Copy code
yaml
    lifecycle:
      preStop:
        exec:
          command: ["/opt/starrocks/be/bin/stop_be.sh", "-g"]
3. 
stop_be.sh
vs 
stop_be.sh -g
 It is critical to use the 
-g
flag. * 
stop_be.sh
(without 
-g
) sends a 
SIGKILL
(immediate death). * 
stop_be.sh -g
sends the 
SIGTERM
that triggers the graceful logic. 4. Configuration Mismatch Even if the signal is received, the BE might not wait long enough for your ingestion task to finish. Check these two configurations in your `be.conf`: * 
loop_count_wait_fragments_finish
: Default is 
2
. The BE waits 
count * 10 seconds
. So by default, it only waits 20 seconds. If your Stream Load takes 30 seconds, it will be cut off. * 
graceful_exit_wait_for_frontend_heartbeat
: Set this to 
true
(available in v3.4.5+). This ensures the BE waits for the FE to acknowledge its "Exiting" status before it actually shuts down. 5. K8S 
terminationGracePeriodSeconds
 By default, Kubernetes only waits 30 seconds before force-killing a pod. * If your BE is trying to be "graceful" and wait for a 60-second load to finish, but K8S kills the pod at 30 seconds, the load will fail. * Fix: Increase 
terminationGracePeriodSeconds: 120
in your Deployment/StatefulSet spec. Summary Checklist for Graceful Ingestion: 1. Version: Use StarRocks v3.4 or higher. 2. Hook: Use a 
preStop
hook with 
stop_be.sh -g
. 3. Timeout: Set 
terminationGracePeriodSeconds
> 
loop_count_wait_fragments_finish * 10
. 4. FE Awareness: Set 
graceful_exit_wait_for_frontend_heartbeat = true
in 
be.conf
. If you meet these criteria, a single BE restarting should allow in-flight loads to finish and tell the FE not to send new loads to that node, resulting in zero-downtime ingestion. References • administration/management/graceful_exit.md • deployment/helm.md • administration/management/BE_configuration.md References • administration/management/graceful_exit.md • quick_start/helm.md
s
graceful_exit_wait_for_frontend_heartbeat
= true works better indeed,
Open in Slack
PreviousNext
Powered by