Hi. Team. We have a real-time table in UPSERT mode...
# troubleshooting
c
Hi. Team. We have a real-time table in UPSERT mode(PINOT 1.2). Sometimes, segment under-replica occurs in this table. During FLUSH process, segment cannot be changed from CONSUMING to ONLINE state. We could see failed messages in Zookeeper instance > Message node. There were accumulated messages for segments created after failed segment. This phenomenon can be normalized only after restarting the server. We could not find EXCEPTION or any WARN, ERROR logs in server log. Are there any similar cases? What should we look at to solve this problem? EX) WORKER-1, TABLE_NAME__1__1__TIMESTAMP, ONLINE WORKER-2, TABLE_NAME__1__1__TIMESTAMP, CONSUMING WORKER-3, TABLE_NAME__1__1__TIMESTAMP, CONSUMING WORKER-1, TABLE_NAME__1__2__TIMESTAMP, CONSUMING
m
You may not have setup the right level of logging or config, any error that could potentially cause this is logged. Most frequent reasons for this to happen: • Segment build failed • Segment commit failed - could be due to access issues to deepstore
c
@Mayank I don't think there is anything wrong with the logging settings. Here are the logging settings.
Copy code
<Loggers>
<Root level="info" additivity="false">
<AppenderRef ref="console"/>
<AppenderRef ref="pinotLog"/>
</Root>
<Logger name="org.apache.helix" level="info" additivity="false">
<AppenderRef ref="pinotHelixLog"/>
</Logger>
<Logger name="org.apache.pinot.server.starter.helix.HelixServerStarter" level="info" additivity="false">
<AppenderRef ref="console"/>
</Logger>
<AsyncLogger name="org.reflections" level="warn" additivity="false"/>
</Loggers>
Is there a retry or timeout setting that can be applied to build & commit failures?
m
Build / commit failures most certainly will show up in the logs.