:wave: Hello, team! Was wondering if anyone can he...
# troubleshooting
l
👋 Hello, team! Was wondering if anyone can helps diagnose an issue we are experiencing with our cluster. We are consuming data from several kafka topics into realtime tables, ingestion stops but we cant seem to find any issue on the logs and also the checks on the tables dont show any issue: Consuming Segments:
Copy code
{
  "serversFailingToRespond": 0,
  "serversUnparsableRespond": 0,
  "_segmentToConsumingInfoMap": {
    "ericsson_ran_enodebfunction__0__16__20250530T0357Z": [
      {
        "serverName": "Server_100.100.25.106_8098",
        "consumerState": "CONSUMING",
        "lastConsumedTimestamp": 1748580550862,
        "partitionToOffsetMap": {
          "0": "465530"
        },
        "partitionOffsetInfo": {
          "currentOffsetsMap": {
            "0": "465530"
          },
          "latestUpstreamOffsetMap": {
            "0": "528919"
          },
          "recordsLagMap": {
            "0": "63389"
          },
          "availabilityLagMsMap": {
            "0": "8"
          }
        }
      }
    ]
  }
}
Table Debug: [ { "tableName": "ericsson_ran_enodebfunction_REALTIME", "numSegments": 5, "numServers": 4, "numBrokers": 2, "segmentDebugInfos": [], "serverDebugInfos": [], "brokerDebugInfos": [ { "brokerName": "Broker_100.100.123.30_8099", "idealState": "ONLINE", "externalView": "ONLINE" }, { "brokerName": "Broker_100.100.57.158_8099", "idealState": "ONLINE", "externalView": "ONLINE" } ], "ingestionStatus": { "ingestionState": "HEALTHY", "errorMessage": "" }, "tableSize": { "reportedSize": "9 MB", "estimatedSize": "9 MB" } } ]
m
Typically, ingestion would stop if there’s no data upstream, or connection issue, or consumer thread dies/stuck during commit. Each one should have some indication in the log. Can you search for the segment name in server log and see what’s happening with it?
l
Hi, thanks for the quick reply. We found the issue just now, we used the default server settings so data was being stored /tmp. I beleive this setting is waht we should change: pinot.server.instance.tierConfigs.hotTier.dataDir