Hello Druids, I am facing an issue where I ingest ...
# generalb
Bhavok
05/02/2024, 1:43 PMHello Druids, I am facing an issue where I ingest data via json(gz) files every day, so one segment is created everyday for a day. For one of the dated segments data is present but I am unable to query it as it shows 0 data for this date. Below is the attached segment description. The segment is showing size as 68 MB & 215,990 rows. Below are the screenshots from DRUID UI console. Can anyone suggest ways to see this data?
a
Amatya Avadhanula
05/02/2024, 2:48 PMThere are two core partitions and this is the first of them
Amatya Avadhanula
05/02/2024, 2:49 PMData for a version will not be queryable until all its core partitions are present.
Amatya Avadhanula
05/02/2024, 2:50 PMDo you see any segment with the above id suffixed with
_1b
Bhavok
05/02/2024, 2:59 PMHi @Amatya Avadhanula , It was intentionally drop as a java job(for ingesting data into druid) ran for two days due to some code errors which caused duplicate data for 18-01-2024, so I thought dropping duplicate segment would work in that case but it didn't
a
Amatya Avadhanula
05/02/2024, 2:59 PMYes dropping a core partition is not advisable.
Is it possible to mark this segment as unused as well and reingest the data?
b
Bhavok
05/02/2024, 3:03 PMYes It is possible for me now, thanks. Can you advise on how can I configure Druid to remove duplicates or just not show them while querying?
a
Amatya Avadhanula
05/02/2024, 3:05 PMWhat kind of duplicates are you expecting?
Amatya Avadhanula
05/02/2024, 3:05 PMb
Bhavok
05/02/2024, 3:29 PMWe are expecting exact duplicates for a day onto next day(ie exact num of rows duplicated 7 ingested next day)