Apache Pinot

I'm going to pin this here for Monday. We are using the realtime to offline segments task and our realtime table has this confiuration for the job:

```  "task": {
    "taskTypeConfigsMap": {
      "RealtimeToOfflineSegmentsTask": {
        "bucketTimePeriod": "1d",
        "bufferTimePeriod": "1d",
        "roundBucketTimePeriod": "1d",
        "mergeType": "rollup",
        "user_raw_risk_score.aggregationType": "max",
        "user_risk_score.aggregationType": "max",
        "user_threat_score.aggregationType": "max",
        "maxNumRecordsPerSegment": "5000000",
        "schedule": "$REALTIME_TO_OFFLINE_SEGMENT_TASK_SCHEDULE"
      }
    }
  }```

```  "segmentsConfig": {
    "timeColumnName": "event_timestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "immutable_events",
    "replicasPerPartition": "$SEGMENTS_CONFIG_REPLICATION",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "$RETENTION_IMMUTABLE_REALTIME_TIME_VALUE_DAYS"
  },```

The variable we pass for prod is 3days retention

What I'm seeing around midnight though and for a few hours after that is this when I run a report by day (our event_timestamp is at the day level)

image.png

Notice the data from 8/4 has 5,293,475 records. So it's selecting the realtime segments there. Even though the offline segments have been rolled up and built, the broker is not choosing the offline data until hours later

This is what I was hoping it would look like seamlessly at midnight.

So maybe I have a config wrong there, but I was thinking retention of 3d would mean the timestamp would be limited to 3d, so I would never get that 4th day  (8/4) of realtime like I did around midnight.

It is not by retention but time boundary on what data is served from where <https://dev.startree.ai/docs/pinot/concepts/time-boundary|https://dev.startree.ai/docs/pinot/concepts/time-boundary>