Node Death caused by SIGSEV

*** SIGSEGV (@0x3) received by PID 27 (TID 0x7dbe405d4640) LWP(852) from PID 3; stack trace: ***
    @     0x7dbf41043ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11a38c48 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7dbf421e8706 PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0]
    @     0x7dbf421e917e JVM_handle_linux_signal
    @     0x7dbf40fec520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0xd36c865 starrocks::SegmentIterator::_switch_context(starrocks::SegmentIterator::ScanContext*)
    @          0xd37e14b starrocks::SegmentIterator::_init_context()
    @          0xd37e99e starrocks::SegmentIterator::_init()
    @          0xd387573 starrocks::SegmentIterator::do_get_next(starrocks::Chunk*)
    @          0xd93e5a8 starrocks::SegmentIteratorWrapper::do_get_next(starrocks::Chunk*)
    @          0xd6428dc starrocks::TimedChunkIterator::do_get_next(starrocks::Chunk*)
    @          0xd6754c0 starrocks::UnionIterator::do_get_next(starrocks::Chunk*)
    @          0xd62f46f starrocks::TabletReader::do_get_next(starrocks::Chunk*)
    @          0xc3c12de starrocks::pipeline::OlapChunkSource::_read_chunk_from_storage(starrocks::RuntimeState*, starrocks::Chunk*)
    @          0xc3c1dcf starrocks::pipeline::OlapChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*)
    @          0xbe7eb6f starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*)
    @          0xabc05fe auto starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::{lambda(auto:1&)#1}::operator()<starrocks::workgroup::YieldContext>(starrocks::workgroup::YieldContext&) const [clone .constprop.0]
    @          0xbdc83be starrocks::workgroup::ScanExecutor::worker_thread()
    @          0xdeda788 starrocks::ThreadPool::dispatch_thread()
    @          0xded1325 starrocks::Thread::supervise_thread(void*)
    @     0x7dbf4103eac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7dbf410d08c0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268bf)

I hit a condition where a tablet gets into a bad state, and then any further

INSERT

statements affecting that tablet cause node death. When this happens, the only remediation I've found is to just delete the table. Is there a better recovery playbook for situations like this?

do you have detailed repro step?

I haven't been able to track down what causes the initial corruption - but I can dump logs of when it gets caught in this death loop if that is helpful? I grabbed some before blowing away the table

checked the log, looks like not very useful, did you still have the logs that just a few minutes before the first ever crash, that could be something. subsequent crash should be the something corrupted already.

It just happened again, dumping whole log file to try and find something more useful

ok.

oh - it looks like logs weren't emitted during the death period 😕

didn't see the crash stack in the log, is it caused by oomkiller?

Things are well well under utilization - it would need to be ~270GB spike. I'll see if I can find any external cause

check the pod event, should tell you the last exit code.

sad, we only retain 1 hour

ok

cc @Murphy Did more testing and discovered its actually queries against the backing data causing the

SIGSEV

- not ingestion. The only thing i've figured out for root cause so far is that disabling flat json, or setting the sparsity higher, fixes the issue. I have tried a bunch of random configuration changes, but it seems like there is something in the shape of our data that is putting things into an invalid state.