https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
j
Do your segment granularities match between the ingestion and reindexing ... and are you explicitly specifying your intervals in the reindex spec for the granularitySpec?
Copy code
"granularitySpec": {
  "type": "uniform",
  "segmentGranularity": "DAY",
  "queryGranularity": "DAY",
  "rollup": true,
  "intervals": ["2023-03-16T00:00:00.000Z/2023-03-17T00:00:00.000Z"],
},
There could be a locking issue if you leave out the intervals or put an empty set [] ... you should specify them both on the input side and the output side of the spec ... specifically this is the case where you are reindexing on a datasource that has actively running stream ingestion.
j
I think so yes, • index_parallel:
Copy code
"granularitySpec": {
                "type": "uniform",
                "segmentGranularity": "MONTH",
                "queryGranularity": "DAY",
                "rollup": true,
                "intervals": [
                    "2023-02-15T00:00:00.000Z/2023-02-22T00:00:00.000Z"
                ]
            },
• kinesis:
Copy code
"granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "MONTH",
        "queryGranularity": "DAY",
        "rollup": true,
        "intervals": []
      },
For the index_parallel :
Copy code
"context": {
        "forceTimeChunkLock": true,
        "useLineageBasedSegmentAllocation": true,
        "priority" : 80
    }
j
Which data is getting "lost"? The reindex data or the data that just streamed in for that time period?
Since you have appendToExisting=false, that's an overwrite of the time period, not an append ... are you running multiple backfills for the same time period, one set of data replaced by the next?
j
I think the data being lost is the one re-indexed
and yeah I do not want to append since it might result in duplicate data
j
What's your task duration and/or intermediate handover period on the Kinesis ingestion spec? Is it something reasonable like an hour? I would think the next thing to try is suspending the Kinesis ingestion while you run a reindex, to make sure the reindex is doing what you want by itself. You do have rollup set to true, so you will be potentially eliminating records in the reindex ... so first make sure that is doing what you want.
j
The 
intermediatePersistPeriod
is set to PT5M
the thing also is that I am indexing or backfilling past data, so even if I do intervals like 
2023-02-15T00:00:00.000Z/2023-03-16T00:00:00.000Z
, why would I need to stop real time ingestion?
j
Not intermediatePersist ... that is just interim disk spills ... I was looking to see how often you are creating new segments ... so 
taskDuration
and 
intermediateHandoffPeriod
. (by default 
intermediateHandoffPeriod
is set very high so that only taskDuration and maxRows/BytesSize are used as the threshold to build new segments ... so 
taskDuration
may be the most relevant setting here. Backfilling much older time chunks that real-time ingestion doesn't get near -- in that case there should be no contention at all between the two. So maybe the issue is that you are doing the rollup, which can reduce record count?
j
ah all right, 
intermediateHandoffPeriod
is 
P2147483647D
and 
taskDuration
is set to 
PT3600S
I do not think the issue is the rollup since I confirm data is gone by querying. I have two cluster, one set up last year where I did not need to run any backfilling and another cluster set up two months or so ago and for that one I needed to backfill some data
j
Okay it looks like your real-time ingestion is set to the default values ... not an issue then. I've never heard of any sort of reindexing or compaction losing data or segments. I don't know if you accidentally ran two reindexing jobs at the same time, if that would be an issue. The reindexing jobs -- are they reingesting data from outside of Druid or are they using existing datasources? One thing you could try is to reindex into a "side" datasource and then compare the data there vs what's in the main datasource during that same time period. Also, a side topic, have you been running auto-compaction to remove the basic segment fragmentation? Since the real-time ingestion bits off a new segment each hour, in a given day you will have a minimum of 24 segments for that day ... if your volumes don't fill each of these segments to the limit then there should be some ability to consolidate segments here.
j
I have run the re-index jobs once at a time. I re-indexed data, checked it by querying the affected intervals and saw that data was there. Some days after or even week some user let me know that they do not see data for certain period, and casually that those periods were the ones I backfilled by ingesting data from S3. Data goes to same datasource realtime ingestion is running on
j
Hmmm ... and your retention rules aren't knocking out any of this data? And on the Druid console it shows segments are "Fully available" for this datasource?
j
yeah fully available
retention should be for 1 year in this datasource
j
I'm kind of at a loss on this one ... if there is no immediate issue with segments or time chunk, it seems odd that they would just "disappear" after a week or so. I don't know enough about the product internals yet to suspect anything else (I'm a SQL/Relational veteran but relatively new to Druid) So my next thought would be to run a report periodically that shows counts by day in the datasource (and/or counts by other relevant dimensions) so you can see immediately when something changes unexpectedly ... and then check logs, etc to see if you can figure out what just happened. And if you have enough disk space available, copy the data into a "backup" datasource on a nightly basis so if something does change you can try to figure out what the delta in the data is. I should probably also ask questions like, what version are you running? And please share your ingestion and reindex specs? etc. Thanks. John
k
Coming from my thread, is there any pattern to the dropped data?
I.e. I found that almost all of my data was dropped in the latter parts of the month by running a series of queries like this
Copy code
SELECT  TIME_EXTRACT(__time, 'YEAR') AS yr, TIME_EXTRACT(__time, 'MONTH') AS mon, TIME_EXTRACT(__time, 'DAY') AS dt, COUNT(*)
FROM "datasource" where __time >= '2022-12-29' and __time < '2023-11-01'
GROUP BY TIME_EXTRACT(__time, 'DAY'), TIME_EXTRACT(__time, 'MONTH'), TIME_EXTRACT(__time, 'YEAR')   ORDER BY yr, mon, dt
That showed me where I had data
Copy code
SELECT  TIME_EXTRACT(__time, 'DAY') AS dt, COUNT(*)
FROM "datasource" where __time >= '2022-12-29' and __time < '2023-11-01'
GROUP BY TIME_EXTRACT(__time, 'DAY')  ORDER BY dt
That showed the pattern within the month: far more data in the first week or so
j
So for the first query, where I have data. 
where __time >= '2023-02-01' and __time < '2023-03-29'
- the amount of data for the last couple of days in Feb (26th) missing, is minimal compared to beginning of March. And then, it seems data is being lost starting March 16th
k
Hmm. The fact that there is a bit of data on the 16th is interesting. Similarly with the 22nd. That's 7 days. I also wonder i the 200K or so records on the 16th are spread throughout the day, or clustered around a certain time period (e.g. the first hour of the day)
j
Exactly, the pattern is hard to follow
Even so, we also have a twin cluster, with exact same configuration, and it has way more data if I run same queries, not only for the missing days, but for existing ones it holds more data
k
That does make the problem being the compaction spec (like mine is) seem a little unlikely.
j
Both compaction specs are identical
I have been tempted to re-index data setting 
dropExisting
to true
basically because I think the issue got triggered when I removed one dimension from the realtime ingestion and I then triggered re-index too without that dimension we did not need anymore
j
I think at this point I would look at the segment list (from Druid console) around these timeframes, try to figure out if the segments for the "missing" times still exist but are overshadowed, looking for overlapping and/or tombstone segments that might be hiding some of the data. I also don't use time_extract() ... I use time_floor() ... don't know if there is a difference, but worth trying time_floor() to get counts and see if there is a difference. Finally, if this is continuing to happen, if you are not quite sure what data is going missing, then if you have enough space in your cluster I suggest copying out the current month that is still good, into a side datasource, and leave it there to use as a reference.
Open in Slack
PreviousNext
Powered by