Frequent errors during compaction ```compaction scheduler cp StarRocks #questions-and-troubleshooting

Frequent errors during compaction: ```compaction_...

Brent Echols

09/27/2025, 12:18 AM

Frequent errors during compaction:

Copy code

compaction_scheduler.cpp:441] Fail to compact tablet 30644991. version=1521 txn_id=14305269 cost=63s : Internal error: invalid pos

Any idea root cause, or what might lead to this?

Brent Echols

09/27/2025, 12:20 AM

Copy code

E20250927 00:15:31.058887 137048907449920 compaction_scheduler.cpp:441] Fail to compact tablet 30644981. version=1521 txn_id=14305439 cost=20s : Internal error: invalid pos
be/src/storage/rowset/plain_page.h:165 seek_to_position_in_page(0)
be/src/storage/rowset/parsed_page.cpp:361 page->_data_decoder->init()
be/src/storage/rowset/scalar_column_iterator.cpp:399 parse_page(&_page, std::move(handle), page_body, footer.data_page_footer(), _reader->encoding_info(), iter.page(), iter.page_index())
be/src/storage/rowset/scalar_column_iterator.cpp:348 _read_data_page(_page_iter)
be/src/storage/rowset/scalar_column_iterator.cpp:290 _load_next_page(&eos)
be/src/storage/rowset/json_column_iterator.cpp:512 func(iter, c.get())
be/src/storage/rowset/segment_iterator.cpp:155 _column_iterators[i]->next_batch(range, col.get())
be/src/storage/rowset/segment_iterator.cpp:1319 _context->read_columns(chunk, range)
be/src/storage/rowset/segment_iterator.cpp:1459 _read(chunk, rowid, chunk_capacity - chunk_start)
be/src/storage/lake/tablet_reader.cpp:286 _collect_iter->get_next(chunk, source_masks)
be/src/storage/lake/vertical_compaction_task.cpp:75 compact_column_group(is_key, i, column_group_size, column_groups[i], writer, mask_buffer.get(), source_masks.get(), cancel_func)

Brent Echols

09/27/2025, 8:29 AM

Opened https://github.com/StarRocks/starrocks/issues/63632

Murphy

09/28/2025, 2:33 AM

do you have a clear procedure to reproduce this bug ?

Murphy

09/28/2025, 2:35 AM

like any specialty of the data ?

Brent Echols

10/13/2025, 10:42 PM

Hey! Apologies - went on vacation 🙂 Is there an easy way to find the offending data? Looking through logs I didn't see anything obvious. This is our prod data so I can't dump the whole DB to you

Murphy

10/14/2025, 3:01 AM

the tablet id is 31035343, 1. you can find the table via

select * from <http://information_schema.be|information_schema.be>_tablets where tablet_id = 31035343

2. then you can try to query the json data in this table, ideally narrow down the record which cause the crash,

select json from t1 tablet(31035343)

Brent Echols

10/14/2025, 8:17 PM

Hmmm - I upgraded to v4 and it seems data is happily loading and compacting now. I'll report back again and see if I can track down the offending data if it happens again

Murphy

10/15/2025, 2:12 AM

sounds good

Open in Slack

Previous Next