https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Team, Pinot Realtime tables connected to Kafka ...
# troubleshooting
m
Hi Team, Pinot Realtime tables connected to Kafka topics stops updating after sometime even though Kafka topics gets populated this is the debug information [ { "tableName": "DISH12_REALTIME", "numSegments": 1, "numServers": 1, "numBrokers": 1, "segmentDebugInfos": [ { "segmentName": "DISH12__0__0__20220817T1159Z", "serverState": { "Server_10.90.21.84_8098": { "idealState": "CONSUMING", "externalView": "CONSUMING", "segmentSize": "0 bytes", "consumerInfo": { "segmentName": "DISH12__0__0__20220817T1159Z", "consumerState": "NOT_CONSUMING", "lastConsumedTimestamp": 1660823957234, "partitionToOffsetMap": { "0": "101151" } }, "errorInfo": { "timestamp": "2022-08-18 115922 UTC", "errorMessage": "Could not build segment", "stackTrace": null } } } } ], "serverDebugInfos": [], "brokerDebugInfos": [], "tableSize": { "reportedSize": "0 bytes", "estimatedSize": "0 bytes" }, "ingestionStatus": { "ingestionState": "UNHEALTHY", "errorMessage": "Segment: DISH12__0__0__20220817T1159Z is not being consumed on server: Server_10.90.21.84_8098" } }
k
Hi, can you check the server logs for 
Could not build segment
stacktrace
m
Yes I have that error in server logs
k
can you share it, generally it can be due to a poison pill, a record that causes error in consumption or some offset getting expired in rare cases
also, what's the pinot version you are using?
m
pinot version is 0.9.0
server.log
there is a error in server logs
java.lang.NullPointerException: null 2022/08/24 043759.411 ERROR [LLRealtimeSegmentDataManager_DISH13__0__0__20220823T0834Z] [DISH13__0__0__20220823T0834Z] Caught exception while transforming the record: { "nullValueFields" : [ "Destination", "RAN-UE-NGAP-ID", "5G-TMSI", "5GMM_cause", "5GSM_cause", "No.", "Info", "Protocol", "Source", "Destination_Port" ], "fieldToValueMap" : { "Destination" : "null", "RAN-UE-NGAP-ID" : "null", "Time" : null, "No." : "null", "Info" : "null", "Source" : "null", "MSIN" : "100000031", "STATUS" : "OK", "5G-TMSI" : "null", "5GMM_cause" : "null", "5GSM_cause" : "null", "Protocol" : "null", "Destination_Port" : "null" } }
even though there is data but it is showing null and failing with null pointer exception
k
This seems to be the one other exception. Either column datatype is set to Long and not String OR the datetime format is set incorrect
Copy code
2022/08/18 11:59:22.606 ERROR [LLRealtimeSegmentDataManager_DISH12__0__0__20220817T1159Z] [DISH12__0__0__20220817T1159Z] Could not build segment
java.lang.NumberFormatException: For input string: "6/13/22 01:35 AM"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:?]
	at java.lang.Long.parseLong(Long.java:692) ~[?:?]
	at java.lang.Long.parseLong(Long.java:817) ~[?:?]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.writeMetadata(SegmentColumnarIndexCreator.java:651) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:603) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:286) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:258) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:815) [pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:746) [pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:644) [pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at java.lang.Thread.run(Thread.java:829) [?:?]
m
As of now it is failing with this error where there is data in the columns but failing with NULL found error
errorInfo" : { "timestamp" : "2022-08-24 043759 UTC", "errorMessage" : "Caught exception while transforming the record: {\n \"nullValueFields\" : [ \"Destination\", \"RAN-UE-NGAP-ID\", \"5G-TMSI\", \"5GMM_cause\", \"5GSM_cause\", \"No.\", \"Info\", \"Protocol\", \"Source\", \"Destination_Port\" ],\n \"fieldToValueMap\" : {\n \"Destination\" : \"null\",\n \"RAN-UE-NGAP-ID\" : \"null\",\n \"Time\" : null,\n \"No.\" : \"null\",\n \"Info\" : \"null\",\n \"Source\" : \"null\",\n \"MSIN\" : \"100000031\",\n \"STATUS\" : \"OK\",\n \"5G-TMSI\" : \"null\",\n \"5GMM_cause\" : \"null\",\n \"5GSM_cause\" : \"null\",\n \"Protocol\" : \"null\",\n \"Destination_Port\" : \"null\"\n }\n}", "stackTrace" : "java.lang.NullPointerException\n"
this is the data
Copy code
{
  "NO": "15",
  "DESTINATION_PORT": "",
  "PROTOCOL": "NGAP/NAS-5GS",
  "TMSI": "",
  "SM_CAUSE": "",
  "TIME": "6/13/22 10:36 PM",
  "INFO": "SACK (Ack=2, Arwnd=106496) , DownlinkNASTransport, Authentication request",
  "MSIN": "100000031",
  "MM_CAUSE": "",
  "STATUS": "OK",
  "DESTINATION": "127.0.0.5",
  "SOURCE": "127.0.0.1",
  "RAN_UE_NGAP_ID": "1"
}
k
Hi it would help if you can share the complete stacktrace. the logs for this date don't seem to be present in the 
server.log
file shared
Open in Slack
PreviousNext
Powered by