https://pinot.apache.org/ logo
Join Slack
Powered by
hi team. Is there anyway to check minion task: `Re...
# troubleshooting
p
hi team. Is there anyway to check minion task: 
RealtimeToOfflineSegmentsTask
status or error message ? I’m find warn log in brokers-server. But cannot find any log information in minion-server. Is this log relations to this task ?
Copy code
2021/03/17 09:26:49.639 WARN [TimeBoundaryManager] [HelixTaskExecutor-message_handle_thread] Failed to find segment with valid end time for table: RuleLogsUAT_OFFLINE, no time boundary generated
2021/03/17 09:27:06.989 WARN [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-0] Failed to find time boundary info for hybrid table: RuleLogsUAT
n
The messages will be either in the controller or minion log. Broker messages will not be related to the minion task
m
The log you posted above happens when the endTime in the segment zk metadata is <= 0:
p
I’m found the errors logs in controller-logs
Copy code
2021/03/17 10:34:53.192 WARN [ZkClient] [TaskJobPurgeWorker-pinot-quickstart] Failed to delete path /pinot-quickstart/PROPERTYSTORE/TaskRebalancer/TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349! org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /pinot-quickstart/PROPERTYSTORE/TaskRebalancer/TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 10:35:07.504 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(a699ebbf_TASK)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 10:35:07.517 ERROR [TaskUtil] [TaskJobPurgeWorker-pinot-quickstart] Job TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349 exists in JobDAG but JobConfig is missing! Job might have been deleted manually from the JobQueue: TaskQueue_RealtimeToOfflineSegmentsTask, or left in the DAG due to a failed clean-up attempt from last purge.
2021/03/17 10:35:07.607 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(fa8a46d1_TASK)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 10:35:07.796 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(329800c1)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 10:35:07.878 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(6e9abcf3_TASK)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.consumer.prop.group.id' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'streamType' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'realtime.segment.flush.segment.size' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/03/17 10:37:53.084 WARN [ConsumerConfig] [pool-8-thread-8] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/03/17 11:04:53.201 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(4edd07a9)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 11:30:46.186 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(597f559b_TASK)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 11:30:46.361 WARN [ZkClient] [TaskJobPurgeWorker-pinot-quickstart] Failed to delete path /pinot-quickstart/PROPERTYSTORE/TaskRebalancer/TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615891980352! org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /pinot-quickstart/PROPERTYSTORE/TaskRebalancer/TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615891980352
2021/03/17 11:30:46.406 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-quickstart-(27f5421b_TASK)] Fail to read record for paths: {/pinot-quickstart/INSTANCES/Server_pinot-server-0.pinot-server-headless.analytics.svc.cluster.local_8098/MESSAGES/49538041-4770-474f-b8b3-a414e638244f=-101}
2021/03/17 11:30:46.486 ERROR [JobDispatcher] [HelixController-pipeline-task-pinot-quickstart-(27f5421b_TASK)] Job configuration is NULL for TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349
2021/03/17 11:30:46.515 ERROR [TaskUtil] [TaskJobPurgeWorker-pinot-quickstart] Job TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615891980352 exists in JobDAG but JobConfig is missing! Job might have been deleted manually from the JobQueue: TaskQueue_RealtimeToOfflineSegmentsTask, or left in the DAG due to a failed clean-up attempt from last purge.
2021/03/17 11:30:53.066 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(aafda2de_DEFAULT)] Event aafda2de_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/03/17 11:30:53.159 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-quickstart-(50b48792_DEFAULT)] Event 50b48792_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/03/17 11:32:55.353 WARN [SegmentCompletionFSM_RuleLogs__0__0__20210316T1130Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_pinot-server-2.pinot-server-headless.analytics.svc.cluster.local_8098 offset=17939 now=1615980775353 start=1615980646028
my Realtime Table Config
Copy code
{
  "REALTIME": {
    "tableName": "RuleLogs_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "replication": "2",
      "replicasPerPartition": "2",
      "timeColumnName": "created_at_days_epoch",
      "schemaName": "RuleLogs"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant",
      "tagOverrideConfig": {}
    },
    "tableIndexConfig": {
      "sortedColumn": [
        "campaign_id",
        "rule_id"
      ],
      "loadMode": "MMAP",
      "invertedIndexColumns": [
        "user_id",
        "device_id"
      ],
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.topic.name": "xxx",
        "stream.kafka.broker.list": "confluent-cp-kafka-headless.kafka.svc.cluster.local:9092",
        "stream.kafka.consumer.prop.group.id": "c1.promotion",
        "stream.kafka.consumer.type": "lowLevel",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
        "realtime.segment.flush.threshold.rows": "0",
        "realtime.segment.flush.threshold.time": "24h",
        "realtime.segment.flush.segment.size": "200M"
      },
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false
    },
    "metadata": {
      "customConfigs": {}
    },
    "quota": {},
    "task": {
      "taskTypeConfigsMap": {
        "RealtimeToOfflineSegmentsTask": {
          "collectorType": "concat",
          "bucketTimePeriod": "1d",
          "bufferTimePeriod": "1h",
          "maxNumRecordsPerSegment": "1000000"
        }
      }
    },
    "routing": {},
    "query": {},
    "ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "device_id",
          "transformFunction": "jsonPathString(extra_data, '$.user_attributes.audience.device_id')"
        },
        {
          "columnName": "service_code",
          "transformFunction": "jsonPathString(extra_data, '$.service_code')"
        },
        {
          "columnName": "event",
          "transformFunction": "jsonPathString(extra_data, '$.event')"
        },
        {
          "columnName": "created_at_days_epoch",
          "transformFunction": "toEpochDays(created_at_ts)"
        }
      ]
    },
    "isDimTable": false
  }
}
n
@Jackie any idea regarding this error? It's coming from the task framework I think?
j
Yes, the error is from Helix
@Phúc Huỳnh Did you delete any task through controller API?
Copy code
2021/03/17 10:35:07.517 ERROR [TaskUtil] [TaskJobPurgeWorker-pinot-quickstart] Job TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1615888367349 exists in JobDAG but JobConfig is missing! Job might have been deleted manually from the JobQueue: TaskQueue_RealtimeToOfflineSegmentsTask, or left in the DAG due to a failed clean-up attempt from last purge.
Also, the 
WARN
from broker means either the offline segment has invalid end time, or there is no offline segment. Can you check if any segment is pushed to the offline table?
p
i don’t delete any task through controller API.
In my scenario. I config my realtime job segments create in 1 days, then i except config segment task convert realtime table to offline table.
here is some segments yesterday & today.
but when i check segments created yesterday, it’s status is OFFLINE. So i think i have problem with realtime segments to offline segments task
Screen Shot 2021-03-18 at 11.36.39 AM.png
but another partition was mark by status ONLINE
n
if you are seeing the segment uploaded, but offline, then the realtimeToOffline is most likely working properly. The problem is happening at the servers. Anything in the server logs about that segment?
p
i found this.
Screen Shot 2021-03-19 at 9.39.04 AM.png
Open in Slack
PreviousNext
Powered by