http://starrocks.io logo
Join Slack
Powered by
Has anyone seen FE followers get stuck in journal ...
# questions-and-troubleshooting
y
Has anyone seen FE followers get stuck in journal replay because external Iceberg/HDFS metadata lookup blocks? We hit this on one FE: 
ReplayedJournalId
stopped advancing, but 
max_journal_id
kept moving. Queries on that FE used stale metadata and failed with 
starlet err Open hdfs file ... No such file or directory
. jstack showed:
Copy code
"replayer" BLOCKED
  at CachingIcebergCatalog.getTable
  at IcebergMetadata.getTable
  at MetadataMgr.getTable
  at AnalyzeMgr.refreshConnectorTableBasicStatisticsCache
  at EditLog.loadJournal   // OP_ADD_EXTERNAL_BASIC_STATS_META
The cache lock was held by a user query thread:
Copy code
"starrocks-mysql-nio-pool-..." TIMED_WAITING
  at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest
  at ... getFileInfo
  at HadoopInputFile.getLength
  at IcebergHiveCatalog.getTable
  at CachingIcebergCatalog$1.load
So replay was indirectly waiting on HMS/HDFS metadata I/O. Wondering if others have seen this, or if there is already a fix/issue for moving external stats cache refresh off the replayer thread.
@Michael Taranov @Rotem Adhoh cc
WDYT about fixing this by moving external stats cache work out of the journal replayer thread? Concrete idea: - Keep replaying the stats meta itself synchronously. - Dispatch only the best-effort cache refresh/expire asynchronously for: - 
OP_ADD_EXTERNAL_BASIC_STATS_META
- 
OP_REMOVE_EXTERNAL_BASIC_STATS_META
- 
OP_ADD_EXTERNAL_HISTOGRAM_STATS_META
- 
OP_REMOVE_EXTERNAL_HISTOGRAM_STATS_META
- Use bounded non-blocking dispatch; if the queue is full, drop/log the cache work rather than block replay. This would keep journal replay independent from HMS/HDFS availability while preserving lazy cache reload on the next query.
m
No need to think too much 🙂 just give AI a spin (with all available for us), open PR and re-review later with the SR masters.
Open in Slack
PreviousNext
Powered by