We ve just experienced another CN crash Multiple nodes went StarRocks #questions-and-troubleshooting

We've just experienced another CN crash. Multiple ...

Dennis Babyak

12/09/2025, 8:38 PM

We've just experienced another CN crash. Multiple nodes went down, at least one reported this:

Copy code

F20251209 17:11:03.446077 139835400345152 scalar_column_iterator.cpp:199] Check failed: ord == _current_ordinal (1340698 vs. 13594)

Seems like it's coming from here

Dennis Babyak

12/09/2025, 8:56 PM

In fact it is seen on 3 nodes at least

Copy code

F20251209 17:11:03.438343 281468877798976 scalar_column_iterator.cpp:199] Check failed: ord == _current_ordinal (549773 vs. 215159629)

Also

Copy code

F20251209 17:11:03.438343 281468877798976 scalar_column_iterator.cpp:199] Check failed: ord == _current_ordinal (549773 vs. 215159629)

They all have the same query ID referenced (as well as other crashed nodes). In fact we've lost ALL compute nodes. The query uses

CAST

array_contains

and a

LEFT SEMI JOIN

Kevin Cai

12/10/2025, 1:27 AM

looks like it reads less data than expected, can you open a github issue and detailed repro step, we will dig into it.

Dennis Babyak

12/10/2025, 9:54 AM

Sure, here it is.

👍 1

Kevin Cai

12/10/2025, 10:12 AM

notice that the cluster is mixed with arm and x86 arch, are they all little endian settings?

Dennis Babyak

12/10/2025, 10:14 AM

they are on whatever AWS EC2 offers

Dennis Babyak

12/10/2025, 10:17 AM

there are currently 8 x86 machines and 2 ARM. We left 2 gravitons to potentially gather more data for another type of CN crashes described here We used to run a mostly Graviton based cluster which seemed to be very unstable (constant CN crashes) and we moved the majority of the nodes to x86 leaving 2 ARM behind to potentially gather more data for the issue. This time the issue affected a mix of ARM and x86 machines

Kevin Cai

12/10/2025, 10:39 AM

Dennis Babyak

12/10/2025, 2:12 PM

@Kevin Cai confirmed - little endian on both types

Copy code

$ lscpu
Architecture:                aarch64
  CPU op-mode(s):            64-bit
  Byte Order:                Little Endian

and

Copy code

$ lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian

Kevin Cai

12/10/2025, 2:55 PM

roger that.

Dennis Babyak

12/11/2025, 11:33 AM

Hi @Kevin Cai We've just had another crash of all compute nodes. This time the exception is different Here is the GutHub issue https://github.com/StarRocks/starrocks/issues/66641 Please note that we've got rig of ARM hosts and are running only on x86 now, 10 nodes in total. Would be great to understand why are we losing the entire compute cluster so often, it affects our operation quite a lot

Open in Slack

Previous Next