This message was deleted.
# generals
Slackbot
04/13/2023, 4:08 AMThis message was deleted.
a
Amatya Avadhanula
04/13/2023, 4:13 AMHi,
The lateMessageRejectionPeriod controls rejection of data before a certain period during streaming ingestion.
You likely need to configure your datasource's retention rules if the data got dropped earlier than expected -> https://druid.apache.org/docs/latest/operations/rule-configuration.html
s
Sergio Ferragut
04/13/2023, 5:02 AMHow long is the task duration? Have you seen any segments published to deep storage?
j
Jasvir Singh Walia
04/13/2023, 5:30 AMHey @Sergio Ferragut Task duration is 1 hour,
Sorry i'm new to druid so by events published to deep storage you mean, events showing in datasource tab correct?
Jasvir Singh Walia
04/13/2023, 5:32 AMHey @Amatya Avadhanula
I've updated the retention rules in datasource and earlier it was default 36 hours. So i've added a new rule for 90 days.
So hopefully this will work now!
Thanks for you suggestion!
👍 1
s
Sergio Ferragut
04/13/2023, 2:23 PMSeems like the retention was the issue. I was just thinking that you may have been only looking at real-time "in-middle-manager" data and you may have had some issue at publishing time which made the data "disappear". Glad you found the answer!
👍 1