Hi I have a realtime table it stopped consuming Kafka sudden Apache Pinot #troubleshooting

Hi! I have a realtime table it stopped consuming K...

Mohamed Emad

07/04/2022, 11:31 PM

Hi! I have a realtime table it stopped consuming Kafka suddenly when I debug the table I found "errorMessage": "Could not build segment"

Mayank

07/04/2022, 11:38 PM

Any other exception or errors? Likely it is not sized correctly

Mohamed Emad

07/04/2022, 11:41 PM

How to debug if it sized correctly or not

Mayank

07/04/2022, 11:41 PM

Did it run out or memory or give any exception stack trace

Mohamed Emad

07/04/2022, 11:43 PM

I checked logs but there is no issues related to memory, There are multiple tables and other tables working correctly

Mayank

07/04/2022, 11:43 PM

Can you paste the log lines around the above error

Mohamed Emad

07/04/2022, 11:46 PM

ERROR [LLRealtimeSegmentDataManager_Table_Name__0__0__20220704T2244Z] [Table_Name__0__0__20220704T2244Z] Could not build segment

Mohamed Emad

07/04/2022, 11:53 PM

Copy code

[
  {
    "tableName": "Table_Name_REALTIME",
    "numSegments": 1,
    "numServers": 3,
    "numBrokers": 2,
    "segmentDebugInfos": [
      {
        "segmentName": "Table_Name__0__0__20220704T2244Z",
        "serverState": {
          "Server_pinot-server-1.pinot-server-headless.default.svc.cluster.local_8098": {
            "idealState": "CONSUMING",
            "externalView": "CONSUMING",
            "segmentSize": "0 bytes",
            "consumerInfo": {
              "segmentName": "Table_Name__0__0__20220704T2244Z",
              "consumerState": "NOT_CONSUMING",
              "lastConsumedTimestamp": 0,
              "partitionToOffsetMap": {
                "0": "2213846"
              }
            },
            "errorInfo": {
              "timestamp": "2022-07-04 22:58:10 UTC",
              "errorMessage": "Could not build segment",
              "stackTrace": null
            }
          },
          "Server_pinot-server-2.pinot-server-headless.default.svc.cluster.local_8098": {
            "idealState": "CONSUMING",
            "externalView": "CONSUMING",
            "segmentSize": "0 bytes",
            "consumerInfo": {
              "segmentName": "Table_Name__0__0__20220704T2244Z",
              "consumerState": "NOT_CONSUMING",
              "lastConsumedTimestamp": 0,
              "partitionToOffsetMap": {
                "0": "2213846"
              }
            },
            "errorInfo": {
              "timestamp": "2022-07-04 22:58:43 UTC",
              "errorMessage": "Could not build segment",
              "stackTrace": null
            }
          },
          "Server_pinot-server-0.pinot-server-headless.default.svc.cluster.local_8098": {
            "idealState": "CONSUMING",
            "externalView": "CONSUMING",
            "segmentSize": "0 bytes",
            "consumerInfo": {
              "segmentName": "Table_Name__0__0__20220704T2244Z",
              "consumerState": "NOT_CONSUMING",
              "lastConsumedTimestamp": 0,
              "partitionToOffsetMap": {
                "0": "2213846"
              }
            },
            "errorInfo": {
              "timestamp": "2022-07-04 23:00:55 UTC",
              "errorMessage": "Could not build segment",
              "stackTrace": null
            }
          }
        }
      }
    ],
    "serverDebugInfos": [],
    "brokerDebugInfos": [
      {
        "brokerName": "Broker_pinot-broker-0.pinot-broker-headless.default.svc.cluster.local_8099",
        "idealState": "ONLINE",
        "externalView": "ONLINE"
      },
      {
        "brokerName": "Broker_pinot-broker-1.pinot-broker-headless.default.svc.cluster.local_8099",
        "idealState": "ONLINE",
        "externalView": "ONLINE"
      }
    ],
    "tableSize": {
      "reportedSize": "0 bytes",
      "estimatedSize": "0 bytes"
    },
    "ingestionStatus": {
      "ingestionState": "UNHEALTHY",
      "errorMessage": "Segment: Table_Name__0__0__20220704T2244Z is not being consumed on server: Server_pinot-server-1.pinot-server-headless.default.svc.cluster.local_8098"
    }
  }
]

Mayank

07/05/2022, 1:04 PM

To close the thread, server log showed exception on why the segment was not built.

Subbu Subramaniam

07/19/2022, 5:07 PM

@Mayank can you post the exception please?

Mayank

07/19/2022, 5:07 PM

I don’t have it. @Mohamed Emad?

Open in Slack

Previous Next