Hello Is there any way to push data from REALTIME table to o Apache Pinot #getting-started

Hello, Is there any way to push data from REALTIME...

Cheguri Vinay Goud

07/25/2022, 10:12 AM

Hello, Is there any way to push data from REALTIME table to offline table when the segments are full, so that there is no stop in consuming records?

saurabh dubey

07/25/2022, 10:20 AM

It's possible via pinot-minion R2O tasks. You can refer to https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows#config

👍 1

Cheguri Vinay Goud

07/25/2022, 10:48 AM

Hi @saurabh dubey, I've tried with below config but servers stopped consumption after 1000 records

Copy code

"task": {
      "taskTypeConfigsMap": {
        "RealtimeToOfflineSegmentsTask": {
          "bucketTimePeriod": "10m",
          "bufferTimePeriod": "0m",
          "roundBucketTimePeriod": "1m",
          "maxNumRecordsPerSegment": "800"
        }
      }
    }

And I dont see any segments created in offline table. Can you please help me with this?

Cheguri Vinay Goud

07/25/2022, 10:50 AM

Here is the OFFLINE table for reference

Copy code

{
  "OFFLINE": {
    "tableName": "ts_demo_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "schemaName": "ts_demo",
      "segmentPushType": "APPEND",
      "replication": "1",
      "segmentPushFrequency": "HOURLY",
      "timeColumnName": "createdDate",
      "allowNullTimeValue": false,
      "replicasPerPartition": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "noDictionaryColumns": [],
      "rangeIndexColumns": [],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
   {
  "OFFLINE": {
    "tableName": "ts_demo_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "schemaName": "ts_demo",
      "segmentPushType": "APPEND",
      "replication": "1",
      "segmentPushFrequency": "HOURLY",
      "timeColumnName": "createdDate",
      "allowNullTimeValue": false,
      "replicasPerPartition": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "noDictionaryColumns": [],
      "rangeIndexColumns": [],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
      "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false
    },
    "metadata": {},
    "quota": {},
    "routing": {},
    "query": {},
    "ingestionConfig": {},
    "isDimTable": false
  }
}   "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false
    },
    "metadata": {},
    "quota": {},
    "routing": {},
    "query": {},
    "ingestionConfig": {},
    "isDimTable": false
  }
}

saurabh dubey

07/25/2022, 11:03 AM

Did you start pinot-minion instance?

saurabh dubey

07/25/2022, 11:04 AM

Also make sure you controller has PinotTaskManager enabled via controller.task.frequencyPeriod

Cheguri Vinay Goud

07/25/2022, 11:07 AM

Minion pod is running and

controller.task.scheduler.enabled=true

property is added to controller

saurabh dubey

07/25/2022, 11:10 AM

Are there any error logs on server? Also just making sure, did you restart the controller after changing the property?

Mark Needham

07/25/2022, 1:23 PM

We wrote some guides showing how to configure it: https://dev.startree.ai/docs/pinot/recipes/real-time-offline-job-automatic-scheduling https://dev.startree.ai/docs/pinot/recipes/real-time-offline-job

👍 2

Neha Pawar

07/26/2022, 2:57 PM

Generally, if you're observing that realtime consumption stops at the same point, it's because of some exception in segment build. Can you check the Pinot server logs for any errors in segment build?

Open in Slack

Previous Next