http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Starrocks cn are pods getting crash...
# questions-and-troubleshooting
n
@Rocky Starrocks cn are pods getting crashed version : 3.3.9
Copy code
*** Aborted at 1781027926 (unix time) try "date -d @1781027926" if you are using GNU date ***
PC: @          0xdaae920 __aarch64_ldadd4_acq_rel
*** SIGSEGV (@0x74502b3538484b) received by PID 27 (TID 0xffff01d40080) from PID 892880971; stack trace: ***
    @     0xffffaea053dc (/usr/lib/aarch64-linux-gnu/libc.so.6+0x853db)
    @          0x9934928 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0xffffafcf1850 ([vdso]+0x84f)
    @          0xdaae920 __aarch64_ldadd4_acq_rel
    @          0x5409f7c starrocks::Schema::Schema(starrocks::Schema*, std::vector<unsigned int, std::allocator<unsigned int> > const&)
    @          0x5417c70 starrocks::ChunkHelper::convert_schema(std::shared_ptr<starrocks::TabletSchema const> const&, std::vector<unsigned int, std::allocator<unsigned int> > const&)
    @          0x72eb7c4 starrocks::lake::ColumnModePartialUpdateHandler::execute(starrocks::lake::RowsetUpdateStateParams const&, starrocks::lake::MetaFileBuilder*)
    @          0x6cb2dfc starrocks::lake::UpdateManager::publish_column_mode_partial_update(starrocks::TxnLogPB_OpWrite const&, long, std::shared_ptr<starrocks::TabletMetadataPB const> const&, starrocks::lake::Tablet*, starrocks::lake::MetaFileBuilder*, long)
    @          0x83eea98 std::_Function_handler<starrocks::Status (), starrocks::lake::PrimaryKeyTxnLogApplier::apply(starrocks::TxnLogPB const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    @          0x83eeda0 starrocks::lake::PrimaryKeyTxnLogApplier::check_and_recover(std::function<starrocks::Status ()> const&)
    @          0x83f18fc starrocks::lake::PrimaryKeyTxnLogApplier::apply(starrocks::TxnLogPB const&)
    @          0x83e9778 starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>)
    @          0x83dcb58 starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
    @          0x8516b10 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::
    @          0x851dae4 starrocks::ThreadPool::dispatch_thread()
    @          0x85143cc starrocks::thread::supervise_thread(void*)
    @     0xffffaea00398 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x80397)
    @     0xffffaea69e9c (/usr/lib/aarch64-linux-gnu/libc.so.6+0xe9e9b)
/opt/starrocks/cn_entrypoint.sh: line 165:    27 Segmentation fault      (core dumped) $STARROCKS_HOME/bin/start_cn.sh $addition_args
✅ 1
r
Based on the stack trace provided, the crash is occurring in the Compute Node (CN) during the 
publish_version
phase for a Primary Key table using Column-mode Partial Updates on an ARM64 (aarch64) architecture. Analysis of the Crash The stack trace highlights a 
SIGSEGV
at 
__aarch64_ldadd4_acq_rel
, which is a low-level atomic addition operation on ARM. This typically happens when the system attempts to update a reference counter (like a 
shared_ptr
) or an atomic status object on an invalid memory address. Key components involved: * Feature: Column-mode Partial Updates in a shared-data (lake) environment. * Function: 
starrocks::lake::ColumnModePartialUpdateHandler::execute
calling 
starrocks::ChunkHelper::convert_schema
. * Context: This happens during the transaction publish phase, where the CN is reconciling metadata for partial updates. Recommended Actions 1. Upgrade to a later 3.3.x Patch Version 3.3.9 is missing several critical fixes for ARM architecture and Primary Key metadata handling that were introduced in later patches. * 3.3.11 included significant performance optimizations and bug fixes for the ARM architecture instruction sets (#55072). * 3.3.18/3.3.19 fixed various crashes related to 
LakePersistentIndex
and metadata concurrency in shared-data mode. Recommendation: Upgrade to the latest stable patch in the 3.3 series (e.g., 3.3.19 or later). There are known issues with earlier 3.3.x versions on ARM that cause segmentation faults during complex schema conversions or high-concurrency partial updates. 2. Workaround: Switch to Row-mode (If feasible) If the crash is blocking production and you cannot upgrade immediately, you can try changing the partial update mode from "column" to "row" for the specific table, although this may increase I/O for tables with many columns:
Copy code
sql
ALTER TABLE <table_name> SET ("partial_update_mode" = "row");
Note: This only applies if your workload can tolerate the performance characteristics of row-mode partial updates. 3. Check for specific Data Types The stack trace mentions 
convert_schema
. This crash can sometimes be triggered by specific column types (like complex types or large strings) being converted during a partial update. Ensure that all columns involved in the partial update have consistent types and that there aren't any illegal null values being passed to 
NOT NULL
columns during the update. Summary This appears to be a known class of stability issues on ARM64 for the shared-data Primary Key engine in early 3.3.x releases. Upgrading to 3.3.19+ is the primary solution to resolve this. If the issue persists after upgrading, please capture the 
cn.out
and 
cn.WARNING
logs along with the core dump and provide them to the StarRocks community or support. References • release_notes/release-3.3.md
n
@Rocky still failing
r
If the CN pods are still crashing after the initial advice, this confirms a deeper issue with the Shared-data (Lake) Primary Key engine on ARM64 architecture, specifically when handling Column-mode Partial Updates. The crash at 
__aarch64_ldadd4_acq_rel
is a failure of an atomic instruction used for reference counting (like 
shared_ptr
). In version 3.3.9, there is a known race condition or memory corruption path in 
ColumnModePartialUpdateHandler
that triggers this when trying to access or convert tablet schemas. To stop the crashes immediately, you must completely suppress the Column-mode partial update feature across the cluster: 1. Force Row-mode Globally Even if you changed the table property, some automated load jobs (like Routine Load or Stream Load) might still be defaulting to "auto" or "column" mode based on session variables. Run this to force the entire cluster into row-mode:
Copy code
sql
-- This disables the crashing code path for all new transactions
SET GLOBAL partial_update_mode = 'row';
*Note: You may need to stop and restart active Routine Loads or Stream Loads to ensure they pick up this global change.* 2. Mandatory Upgrade (Fixes specific to ARM64) Version 3.3.9 has several identified "Segment Fault" bugs on ARM64/Graviton architectures. • Recommendation: Upgrade to at least v3.3.22. • Reason: v3.3.22 contains a critical fix for 
SIGSEGV
crashes in the CN specifically on ARM64 architectures (Issue #66864) and fixes for PK tablet corruption/crashes in column-mode partial updates (Issue #67364). 3. Verify Deployment Environment If you are running on Kubernetes (K8s): • Image Architecture: Ensure you are using the official 
arm64
image tag (e.g., 
starrocks-cn-ubuntu:3.3.22-arm64
). If you accidentally use an 
amd64
image, it may run via emulation (QEMU), which is extremely unstable for atomic memory operations. • Memory Limits: The stack trace shows the crash during 
publish_version
. In the Lake engine, this phase can require a sudden burst of memory to load the Primary Key index. If your pod has a strict 
memory limit
in K8s, it might be getting OOM-Killed or corrupting memory right before the crash. Check 
kubectl describe pod
for 
Terminated: OOMKilled
. 4. Temporary Configuration Bypass If you cannot upgrade immediately, you can attempt to reduce the concurrency of the metadata updates on the CN by adding this to your `cn.conf`:
Copy code
properties
# Reduce the number of threads performing schema updates
update_schema_worker_count = 1
And then restart the CN pods. This reduces the chance of the race condition but does not fix the underlying bug. Summary: The crash is a confirmed bug in the 3.3.x series' handling of ARM atomic instructions during partial updates. Setting 
SET GLOBAL partial_update_mode = 'row';
is the only configuration-side fix, but a move to v3.3.22+ is required for a permanent stability. References • best_practices/query_tuning/schema_tuning.md • administration/management/monitoring/metrics.md • knowledge/trouble_shooting/fe_deadlock_problems.md • integrations/smt/starrocks_sqlalchemy.md • sql-reference/data-types/semi_structured/VARIANT.md • administration/management/BE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/UPDATE.md
n
@Rocky * Aborted at 1781032214 (unix time) try "date -d @1781032214" if you are using GNU date * PC: @ 0xdaae920 __aarch64_ldadd4_acq_rel I20260609 191014.410840 281467735179392 starlet.cc:85] Proactively add shard 127215405 back to worker ... * SIGBUS (@0x7267476f34566d) received by PID 28 (TID 0xfffe4fde0080) from PID 1865700973; stack trace: * @ 0xffffb41853dc (/usr/lib/aarch64-linux-gnu/libc.so.6+0x853db) @ 0x9934928 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) @ 0xffffb5471850 ([vdso]+0x84f) @ 0xdaae920 __aarch64_ldadd4_acq_rel @ 0x5409f7c starrocks:Schema:Schema(starrocks::Schema*, std::vector<unsigned int, std::allocator<unsigned int> > const&) I20260609 191014.444845 281470240620672 vertical_compaction_task.cpp:99] Vertical compaction finished. tablet: 131380893, txn_id: 296170078, statistics: {"read_local_sec":0,"read_local_mb":0,"read_remote_sec":1,"read_remote_mb":2,"read_remote_count":370,"read_local_count":0,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":7,"pk_sst_merge_sec":0} @ 0x5417c70 starrocks:ChunkHelper:convert_schema(std:shared ptr&lt;starrocks:TabletSchema const> const&, std::vector<unsigned int, std::allocator<unsigned int> > const&) @ 0x72eb7c4 starrocks:lakeColumnModePartialUpdateHandler:execute(starrocks:lake:RowsetUpdateStateParams const&, starrocks:lake:MetaFileBuilder*) @ 0x6cb2dfc starrocks:lakeUpdateManager:publish_column_mode_partial_update(starrocks::TxnLogPB_OpWrite const&, long, std:shared ptr&lt;starrocks:TabletMetadataPB const> const&, starrocks:lake:Tablet*, starrocks:lake:MetaFileBuilder*, long) I20260609 191014.490082 281467848425600 starlet.cc:85] Proactively add shard 271990721 back to worker ... I20260609 191014.490356 281467735179392 starlet.cc:85] Proactively add shard 127215406 back to worker ... @ 0x83eea98 std: Function handler&lt;starrocks:Status (), starrocks:lakePrimaryKeyTxnLogApplier:apply(starrocks::TxnLogPB const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) I20260609 191014.505576 281467839971456 starlet.cc:85] Proactively add shard 251622979 back to worker ... @ 0x83eeda0 starrocks:lakePrimaryKeyTxnLogApplier:check_and_recover(std:function&lt;starrocks:Status ()> const&) @ 0x83f18fc starrocks:lakePrimaryKeyTxnLogApplier:apply(starrocks::TxnLogPB const&) @ 0x83e9778 starrocks:lake:publish_version(starrocks:lake:TabletManager*, long, long, long, std:span&lt;starrocks:TxnInfoPB const, 18446744073709551615ul>) @ 0x83dcb58 starrocks:LakeServiceImpl:publish_version(google:protobuf:RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google:protobuf:Closure*)::{lambda()#1}::operator()() const @ 0x8516b10 std::_Function_handler<void (), starrocks:ConcurrencyLimitedThreadPoolToken:submit(std:shared ptr&lt;starrocks:Runnable>, std:chronotime point&lt;stdchrono V2:system_clock, std:chrono:duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>:: @ 0x851dae4 starrocks:ThreadPool:dispatch_thread() I20260609 191014.572864 281467735179392 starlet.cc:85] Proactively add shard 127215407 back to worker ... @ 0x85143cc starrocks:🧵:supervise_thread(void*) @ 0xffffb4180398 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x80397) @ 0xffffb41e9e9c (/usr/lib/aarch64-linux-gnu/libc.so.6+0xe9e9b) /opt/starrocks/cn_entrypoint.sh: line 165: 28 Bus error (core dumped) $STARROCKS_HOME/bin/start_cn.sh $addition_args root@ip-10-2-34-75:/home/ubuntu#
r
The new trace provides a critical piece of information: the crash is now a 
SIGBUS
(Bus error) at the memory address 
0x7267476f34566d
. Why this is happening On ARM64 (aarch64) architecture, atomic instructions like 
__aarch64_ldadd4_acq_rel
(used for internal reference counting and metadata updates) strictly require 4-byte memory alignment. The address in your log (
...6d
) is not divisible by 4. This confirms a code-level logic bug in StarRocks 3.3.9 where an unaligned or corrupted memory pointer is being passed to an atomic operation during the schema conversion of a Partial Update. This is a known stability issue for the shared-data (Lake) engine on ARM in earlier 3.3 releases. Immediate Action Plan 1. Identify and Fix the Table/Job The global 
SET GLOBAL partial_update_mode = 'row';
may not have stopped the crash if your table or loading job has the property explicitly hardcoded. * Check the Table: Run 
SHOW CREATE TABLE <table_name>;
for the table receiving updates. If it has 
"partial_update_mode" = "column"
, you must change it:
Copy code
sql
    ALTER TABLE <table_name> SET ("partial_update_mode" = "row");
* Check the Load Job: If you are using Stream Load, Routine Load, or Flink/Kafka Connectors, check the job configuration. Look for 
partial_update_mode: column
and change it to 
row
. _ _Note: In v3.3.9, "auto" mode might still choose "column" for some tables. Forcing "row" is the only way to avoid this specific code path.* 2. Disable Vertical Compaction (Temporary Workaround) Your logs show 
Vertical compaction finished
just milliseconds before the crash. Vertical compaction and Column-mode Partial Updates share similar schema conversion logic. To stabilize the CN while you plan an upgrade, disable vertical compaction in your `cn.conf`:
Copy code
properties
# Add to cn.conf and restart CN pods
enable_vertical_compaction = false
3. Mandatory Upgrade to v3.3.22+ Version 3.3.9 is nearly a year old and lacks critical ARM64 alignment and memory safety fixes. * Fix #66864: Specifically addresses `SIGSEGV`/`SIGBUS` on ARM64/Graviton architectures. * Fix #66139: Fixes ingestion failures and crashes when column-mode Partial Updates are used. * Recommendation: Move to v3.3.22 or higher. The community has resolved these specific ARM atomic instruction bugs in later 3.3.x patches. Summary Checklist 1. Stop all Load Jobs currently hitting the cluster to stop the crash loop. 2. Explicitly set 
partial_update_mode = row
on all Primary Key tables. 3. Upgrade to 3.3.22 to fix the underlying ARM alignment bug. 16/LSE alignment bug. 4. Verify Image: Ensure you are using the official 
-arm64
tagged Docker image to avoid emulation issues. References • administration/management/BE_configuration.md
n
@Rocky arm64 is not presnet
r
Your logs definitively show that you are running on ARM64 (aarch64) architecture, as evidenced by the 
__aarch64_
instructions and the library path 
/usr/lib/aarch64-linux-gnu/
. If you are saying that ARM64 is "not present" in the context of trying to upgrade or find images, here is the explanation and how to fix this: 1. Where to find ARM64 images StarRocks does not always provide separate 
.tar.gz
binary downloads for ARM64 on the official website (those are usually 
amd64
only). However, the Docker images are multi-architecture. * Docker Hub: If you use 
starrocks/cn-ubuntu:3.3.22
, Docker will automatically pull the 
arm64
version if your host is ARM-based. * Checking your architecture: Run 
uname -m
on your node. If it says 
aarch64
, you are on ARM (e.g., AWS Graviton). 2. Why it is crashing (The "Alignment" Bug) The crash 
__aarch64_ldadd4_acq_rel
is an atomic addition instruction that must be 4-byte aligned. * In version 3.3.9, there is a bug in the 
Schema
conversion logic (specifically in 
ColumnModePartialUpdateHandler
) that passes an unaligned memory address to this instruction. * This triggers a Bus error or SIGSEGV only on ARM64 hardware because x86 is more forgiving of unaligned access. 3. Solution: Use the 3.3.22+ Image A specific fix for ARM64 stability (Issue #66864) was included in later 3.3.x patches. Since you are on K8s/Pods, update your deployment to use: 
image: starrocks/cn-ubuntu:3.3.22
(or a later 3.3.x version). 4. If you cannot find the image or it fails to pull: If your container registry is not pulling the ARM64 version, you can explicitly try to force the platform in your K8s spec or pull command:
Copy code
bash
docker pull --platform linux/arm64 starrocks/cn-ubuntu:3.3.22
5. Immediate Workaround (No Upgrade) If you cannot upgrade today, you must stop the Column-mode partial updates which are triggering the unaligned memory access: 1. Change the table property: 
ALTER TABLE <table_name> SET ("partial_update_mode" = "row");
2. Global switch: 
SET GLOBAL partial_update_mode = 'row';
3. Disable Vertical Compaction in `cn.conf`: 
enable_vertical_compaction = false
Summary: The crash is a known alignment bug on ARM64 in version 3.3.9. It is fixed in v3.3.22. Use the official Docker image to get the ARM64-compatible version. References • developers/build-starrocks/Build_in_docker.md • deployment/prepare_deployment_files.md • release_notes/release-3.3.md
The reason you are seeing "arm64 is not present" on the StarRocks Download page is that StarRocks currently only provides pre-compiled tarball (.tar.gz) packages for x86 (amd64) on the website. However, ARM64 (aarch64) is fully supported via Docker. To run on ARM64 or get the ARM64 binaries, you must use the Docker images. 1. Where to find ARM64 Images You can pull the official ARM64 images from Docker Hub. They are multi-arch, meaning 
docker pull
will automatically grab the 
arm64
version if you are on an ARM host (like AWS Graviton). * For Compute Node (CN): 
starrocks/cn-ubuntu:3.3.9
(or newer) * For Backend (BE): 
starrocks/be-ubuntu:3.3.9
* Artifacts image (to extract binaries): 
starrocks/artifacts-ubuntu:3.3.9
2. How to get ARM64 Binaries (if not using Docker) If you need the 
.tar.gz
files for a manual deployment on ARM64, you have to extract them from the artifacts image:
Copy code
bash
# This pulls the image and extracts the ARM64 release files to your current directory
docker run --rm starrocks/artifacts-ubuntu:3.3.22 \
    tar -cf - -C /release . | tar -xvf -
3. Why it is "Still Failing" (The SIGBUS Error) The 
SIGBUS
error at address 
...6d
in your logs is a memory alignment bug specifically affecting ARM64 in StarRocks v3.3.9. * The Cause: On ARM, certain atomic instructions (like 
ldadd
) require memory to be aligned to 4 or 8 bytes. Version 3.3.9 has a bug where it attempts these operations on unaligned memory during "Column-mode Partial Updates." * The Fix: This was fixed in later patches. You must upgrade to v3.3.22 or higher to resolve this specific ARM alignment crash. Immediate Mitigation (Before Upgrade) To stop the crash immediately without upgrading, you must disable the code path causing the unaligned memory access: 1. Global Variable:
Copy code
sql
    SET GLOBAL partial_update_mode = 'row';
2. Table Property: Check if any table has 
partial_update_mode
set to 
column
and change it:
Copy code
sql
    ALTER TABLE <table_name> SET ("partial_update_mode" = "row");
3. CN Config (
cn.conf
): Disable vertical compaction, as it triggers similar logic:
Copy code
properties
    enable_vertical_compaction = false
Summary: The ARM64 binaries exist but are only distributed via Docker Hub. The crashes you are seeing are confirmed bugs in v3.3.9 on ARM; upgrading to v3.3.22 using the Docker artifacts is the only permanent solution.
n
@Rocky same issue
r
The reason you are seeing the "same issue" (the 
SIGBUS
crash at 
__aarch64_ldadd4_acq_rel
) despite changing settings is that existing transactions or rowsets already created in "column-mode" are still being processed by the background threads (Compaction and Publish Version). Because the bug is a physical memory alignment error in the 3.3.9 code (a 4-byte atomic instruction hitting a non-aligned address), configuration changes often cannot "hide" the faulty code path once it is triggered. 1. How to get the ARM64 Fix (since it's "not present" on the site) The StarRocks website only lists x86 binaries. To get the 3.3.22 ARM64 binaries containing the alignment fix, you must extract them from the official Docker artifacts image. Run this on any machine with Docker:
Copy code
bash
# Pull the 3.3.22 artifacts image (ARM64 version)
docker pull starrocks/artifacts-ubuntu:3.3.22

# Extract the binaries to your local directory
docker run --rm starrocks/artifacts-ubuntu:3.3.22 \
    tar -cf - -C /release . | tar -xvf -
This will give you a 
bin
and 
lib
folder compatible with your ARM64/Graviton instances. This is the only way to permanently stop the 
SIGBUS
crash. ────────── 2. Immediate Stop-Gap (to stop the CN crash loop) If you cannot upgrade yet, you must stop the background processes that are triggering the crash on existing data. A. Disable Column-Mode logic in 
cn.conf
 Add this to your 
cn.conf
(or 
be.conf
) and restart the CN. This is a "kill switch" that forces the engine to bypass the optimized (but crashing) column-mode logic:
Copy code
properties
# Add this to cn.conf
enable_pk_column_partial_update = false
B. Stop Vertical Compaction Your trace showed vertical compaction running right before the crash. Disable it:
Copy code
properties
# Add this to cn.conf
enable_vertical_compaction = false
C. Clean up the "Stuck" Transaction The stack trace shows the crash during 
publish_version
for transaction 
296170078
on tablet 
131380893
. If the CN crashes every time it tries to "publish" this specific data, the transaction may be corrupted due to the alignment bug. 1. Identify the table: 
SELECT * FROM information_schema.be_tablets WHERE tablet_id = 131380893;
2. If the crash persists, you may need to TRUNCATE or DROP that specific table to clear the metadata queue that is causing the CN to reboot-loop. Summary The crash is a confirmed AArch64 memory alignment bug in v3.3.9. 1. Download v3.3.22 via the 
docker run
command above (since the website is x86-only). 2. Set 
enable_pk_column_partial_update = false
 in 
cn.conf
to bypass the buggy code until you can swap the binaries. References • administration/management/BE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/UPDATE.md • deployment/environment_configurations.md • developers/mac-compile-run-test.md • loading/Load_to_Primary_Key_tables.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • faq/shared_data_faq.md • release_notes/release-2.5.md • administration/management/monitoring/metrics.md • knowledge/trouble_shooting/fe_deadlock_problems.md • release_notes/release-3.4.md
n
@Rocky can i point same cn storage to another starrocks cluster?
r
In the Shared-data (Cloud-native) architecture you are using, the answer depends on what you mean by "another cluster." 1. Can I point a new, separate StarRocks cluster (new FEs) to the same storage path? No. You should never do this. * Missing Metadata: The Object Storage (S3/HDFS) only contains the raw data files. The "intelligence" (which file belongs to which table, schema definitions, and transaction logs) lives in the FE metadata (BDBJE). A new cluster will not see your existing tables. * Data Corruption: Each cluster has a "Vacuum" (Garbage Collection) process. If Cluster B doesn't know about Cluster A's files, it will eventually see them as "unreferenced" and delete them from S3, destroying Cluster A's data. 2. Can I have multiple sets of CNs pointing to the same data? Yes. This is called Multi-Warehouse (or Compute Groups). * In StarRocks v3.2+, you can create multiple "Compute Groups." * You can have one group of CNs for "Loading" and another group for "Queries." * They all share the same FE and the same storage. This is the correct way to isolate workloads or test new hardware without moving data. 3. Can I use the same S3 Bucket? Yes, as long as you use different root paths. * Cluster A: 
<s3://my-bucket/cluster-prod/>
* Cluster B: 
<s3://my-bucket/cluster-test/>
* This is perfectly safe. 4. Can I move my CNs to a different cluster and keep their "Storage" (Cache)? No. * The storage on a CN is just a local cache of the data on S3. * This cache is indexed by Tablet IDs and Version IDs generated by the original FE. * A different FE cluster will generate different IDs. If you point these CNs to a new cluster, the existing cache becomes useless "garbage" and will be overwritten. ────────── Recommendation for your current Crash Issue: If you are asking this because your current cluster is stuck in a crash loop (due to the 
SIGBUS
bug we discussed): 1. Do not start a new cluster against that S3 path. You will lose data. 2. Upgrade the current cluster's binaries. Since the crash is a code bug in 
starrocks_be
(the CN process), simply replacing the 
3.3.9
binaries with 
3.3.22
binaries and restarting the CNs will fix the issue. The metadata in the FE will remain intact, and your data will be safe. To recover without data loss: 1. Stop all CN nodes. 2. Replace the CN binaries with v3.3.22 (extract them from the Docker image as shown previously). 3. Restart the CNs. They will reconnect to the existing FE and resume processing without the "Publish Version" tasks without crashing. References • faq/shared_data_faq.md • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • deployment/shared_data/s3.md
2 Views
Open in Slack
PreviousNext
Powered by