Hello team quick doubt regarding data retention I have inges Apache Pinot #general

Hello team, quick doubt regarding data retention. ...

Rohit Anilkumar

01/27/2023, 2:53 PM

Hello team, quick doubt regarding data retention. I have ingested batch data into pinot. And it has data from 2022 Jan 1st till Dec 2022 31st. The retention was set to 365 days. I ingested the data last week( around 18th Jan 2023). I can still see data beyond 18th Jan 2022 in the pinot table. How does retention period works- is it from the date of ingestion or the time column in the data? If it had been the latter, i should not possibly be seeing data lesser than 18th Jan 2022. Can someone help me understand how the retention period works?

Kartik Khare

01/27/2023, 3:21 PM

Hi, it works on the basis of

timeColumn

configured in the table config. The retention job runs periodically in the controller every 6 hours by default. you can check the

endTime

value in the metadata of segments which have data before 18th jan

Rohit Anilkumar

01/27/2023, 3:43 PM

Let me check that and get back to you

Rohit Anilkumar

01/27/2023, 7:05 PM

How can i find the segemnts that cater to data before 18th Jan. I see segements but how do I know which belongs to which date range? Just ran a query and I can still see data prior to Jan 18th.

Copy code

select "time",
  DATETIMECONVERT(
    "time", '1:MILLISECONDS:EPOCH', '1:HOURS:SIMPLE_DATE_FORMAT:yyyy-MM-dd HH:mm', 
    '1:DAYS'
  ) AS convertedTime 
from 
  qrcodes order by "time" asc

time	convertedTime
1640925981801	2021-12-31 00:00
1640926947102	2021-12-31 00:00
1640958221555	2021-12-31 00:00
1640958221562	2021-12-31 00:00
1640958342132	2021-12-31 00:00
1640958342153	2021-12-31 00:00
1640995223313	2022-01-01 00:00
1640995223424	2022-01-01 00:00

Rohit Anilkumar

01/27/2023, 7:06 PM

I have specified 365 days as retention.

Copy code

"segmentsConfig": {
      "timeType": "MILLISECONDS",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "365",
      "replication": "2",
      "timeColumnName": "time",
      "minimizeDataMovement": false
}

Kartik Khare

01/27/2023, 7:51 PM

Copy code

select "time",
  DATETIMECONVERT(
    "time", '1:MILLISECONDS:EPOCH', '1:HOURS:SIMPLE_DATE_FORMAT:yyyy-MM-dd HH:mm', 
    '1:DAYS'
  ) AS convertedTime, $segmentName 
from 
  qrcodes order by "time" asc

Kartik Khare

01/27/2023, 7:51 PM

The timeColumn seems right to me

Rohit Anilkumar

01/27/2023, 8:04 PM

Yea but im seeing data that has crossed the retention period. Thats where im confused

Kartik Khare

01/27/2023, 8:20 PM

Data doesn't matter, the endTime of that segment matters. It could be that there are some records in those segment of time within the retention period. Hence, it didn't get removed.

2 Views

Open in Slack

Previous Next