This message was deleted.
# generals
Slackbot
10/27/2023, 3:05 PMThis message was deleted.
g
Gian Merlino
10/27/2023, 3:28 PMif you're reading from kafka/kinesis, late records are handled by appending new segments to older time chunks. It's automatic
Gian Merlino
10/27/2023, 3:29 PMHandling the late records with batch ingestion is more useful if you need to correct (i.e. replace/overwrite) records that came initially, like a restatement or update
Gian Merlino
10/27/2023, 3:29 PMIf it's just appends then you should be OK with streaming only
s
sriramdas sivasai
10/27/2023, 3:31 PMit takes lot of time to append. I'm suspecting its may try to find the same record in the existing segments and create new one if not found.
is it ?
g
Gian Merlino
10/27/2023, 3:44 PMnah, it always just creates a new segment
Gian Merlino
10/27/2023, 3:44 PMwhich part takes a long time?
s
sriramdas sivasai
10/27/2023, 4:02 PMwe observe the handoff and sometimes its throwing the errors like
couldnt not allocate segment for row with timestamp[2023-10-16T18:37:27.000z]sriramdas sivasai
10/27/2023, 4:03 PMI do observe somequeries being fire on the database druid_segments table
sriramdas sivasai
10/27/2023, 4:03 PMsomething like this
Copy code
20 | druid | 192.168.39.62:48746 | druid | Query | 16 | Sending data | SELECT payload FROM druid_segments WHERE used = 1 AND dataSource = 'data_table' AND ((start < '2023-08-20T00:34:00.001Z' AND `end` > '2023-08-20T00:34:00.000Z') OR (start = '-146136543-09-08T08:23:32.096Z' AND "end" != '146140482-04-24T15:36:27.903Z' AND "end" > '2023-08-20T00:34:00.000Z') OR (start != '-146136543-09-08T08:23:32.096Z' AND "end" = '146140482-04-24T15:36:27.903Z' AND start < '2023-08-20T00:34:00.001Z') OR (start = '-146136543-09-08T08:23:32.096Z' AND "end" = '146140482-04-24T15:36:27.903Z')) |g
Gian Merlino
10/27/2023, 5:14 PMwhat's your
segmentGranularityGian Merlino
10/27/2023, 5:15 PMif you're using a very fine one (like
minutehours
sriramdas sivasai
10/27/2023, 5:16 PMwe do have granularity as an hour
b
Ben Krug
10/27/2023, 11:16 PMIs it possible that compaction is running on the older segments at the same time? I wonder if that could cause problems getting a lock on the time chunk?
s
sriramdas sivasai
10/27/2023, 11:17 PMwe didnt configured the compaction yet.
b
Ben Krug
10/27/2023, 11:27 PMI don't know then... Do all segments have the same granularity, and same as ingestion spec? Also, does it retry and recover later?
s
sriramdas sivasai
10/27/2023, 11:55 PMyes. but it creates huge lag
sriramdas sivasai
10/28/2023, 2:43 AM@Ben Krug @Gian Merlino May i know, how can i do the batch ingestion of this late data to same datasource ? I think, i should be creating the
late_kafka_topiclate_datasourcelate_datasourceoriginal_datasourcej
John Kowtko
10/29/2023, 7:05 PMI wonder if HOUR granularity is still too fine. How late is the data that is arriving? Let's say your 3b late records are up to 30 days old, evenly distributed across those 30 days. with HOUR granularity you are opening up 30 x 24 = 720 time chunks for ingestion, and eventually segment building. That is a lot of segments to build and publish whenever the tasks roll over.
And this is per task ... multiple this times the number of ingestion tasks that you have.
What is your taskDuration set to? If the default (1 hr) then this is happening hourly. Possibly this is contributing to the issues?
One way to tell hoe many time chunks are open is to look at the Segments tab on webconsole, sort by smallest size first, you will see "(realtime)" show up for each open realtime segment (time chunk) ... if you see dozens or hundreds of these, then I would suggest changing your segment granularity to DAY.
b
Ben Krug
10/31/2023, 1:34 AM+1 - and also about what you asked, I think a batch job with
appendToExisting: true2 Views