https://pinot.apache.org/ logo
Join Slack
Powered by
Hello, I have an OFFLINE table with a configured r...
# troubleshooting
e
Hello, I have an OFFLINE table with a configured retention:
Copy code
"segmentsConfig": {
      "replication": "1",
      "timeColumnName": "timestamp",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "90",
      "timeType": "MILLISECONDS",
      "schemaName": "schema_607ec74cd839000300105f34"
    }
But my segments aren’t deleted, like this one:
Copy code
{
  "segment.offline.download.url": "<s3://pinot-production-store/datasource_607ec74cd839000300105f34/datasource_607ec74cd839000300105f34_1618921448331_1618947468383_1>",
  "segment.start.time": "1618921448331",
  "segment.time.unit": "MILLISECONDS",
  "segment.end.time": "1618947468383",
  "segment.total.docs": "390000",
  "segment.table.name": "datasource_607ec74cd839000300105f34",
  "segment.creation.time": "1619049989622",
  "segment.name": "datasource_607ec74cd839000300105f34_1618921448331_1618947468383_1",
  "segment.index.version": "v3",
  "segment.offline.refresh.time": "-9223372036854775808",
  "segment.type": "OFFLINE",
  "segment.offline.push.time": "1619050008800",
  "segment.crc": "1089143347"
}
I see in the logs the following message:
Copy code
2021/09/17 03:45:12.752 INFO [RetentionManager] [pool-9-thread-6] Start managing retention for table: datasource_607ec74cd839000300105f34_OFFLINE
And except this log, no other log message mention the RetentionManager Do you know why or how I can debug this? Thank you
m
Are no segments getting deleted or just a few? If none are getting deleted, perhaps a config issue.
Also, does the controller have the correct permissions to delete from S3 buckets?
And if the segment deletion is successful, you should see something like 
Deleting {} segments from table: {}"
in the log
e
No segments getting deleted from what I see And I don’t see any other logs about the retention manager
m
That seems to indicate that retention manager does not think the segments should be deleted, which points to some misconfiguration (time unit). But from your configs above, I don't see an issue
e
yep I thought the same, and in my online tables, segments are getting deleted
Copy code
"segmentsConfig": {
      "replication": "1",
      "timeType": "MILLISECONDS",
      "timeColumnName": "timestamp",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "3",
      "segmentPushFrequency": "HOURLY",
      "segmentPushType": "APPEND",
      "replicasPerPartition": "1",
      "completionConfig": {
        "completionMode": "DOWNLOAD"
      },
      "schemaName": "schema_607ec7511360000300516e44"
    },
Maybe it’s because it’s an hybrid table?
In my offline table I don’t have the
Copy code
"segmentPushType": "APPEND",
But this is the default value isn’t it?
m
No, hybrid table should still have retention
Is the retention config from offline table or real-time
e
1st config from the offline
2nd one from the online (which works)
found it, without 
"segmentPushType": "APPEND",
it doesn’t work
adding it to my OFFLINE table fixed it
m
I see, thanks for sharing. I’ll add it to the FAQ.
Open in Slack
PreviousNext
Powered by