https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
d
First thing first, is the data available on deep storage? (S3/HDFS)
g
i am not sure what's going on; i suspect it's something related to the "slightly older data in to weeks and even older in to months" piece a potential issue here is that weeks and months are non-aligned: one week can span two different months
k
It has been deleted from deep storage
At least as far as I can tell. In deep storage it has one segment for that range.
And that lines up with the one it has loaded.
Yeah, I was wondering about the lack of alignment as well
g
as to data recovery— do you have autokill enabled? if not then the segments would still be on deep storage somewhere (perhaps in another directory). you could check the 
druid_segments
table of your metadata store to find pointers to them. they'd be there with 
used = false
if they haven't been deleted by autokill (or manual kill)
k
We don't have autokill enabled? (as far as I know). We do periodically submit kill tasks though.
g
if you haven't done one for the relevant time range yet, then the data should still be there somewhere; if you want to check then look in 
druid_segments
for the 
used = false
ones (they're the ones that kill will kill when run)
k
Good idea. I was going to turn off the kill task today, which should give us insight over the next few days
Either way, I'll takea look now
g
as to what happened— mind sharing the 
week
and 
month
compaction specs that you used for the December and January data? and what version you're on?
hopefully, that's enough to analyze or repro what you're seeing
if you have steps to repro on a fresh install that's even better
k
Hmmm. My schema doesn't quite have a used=false column. This is what the web-console suggested:
Copy code
SELECT
  datasource,
  dimensions,
  "end",
  is_available,
  is_overshadowed,
  is_published,
  is_realtime,
  last_compaction_state,
  metrics,
  num_replicas,
  num_rows,
  partition_num,
  segment_id,
  shard_spec,
  size,
  "start",
  version
FROM sys.segments
g
ah, that's the 
sys.segments
virtual metadata table
I meant the solid 
druid_segments
table in your metadata store (mysql or postgresql usually)
k
Alright, thanks. We do have some data in there for the last week or so. A bunch of hourly segments.
I'll look at getting the specs
Here the last set we sent:
Copy code
{"type": "compact", "dataSource": "my-datasource", "interval": "2023-03-18T00:00:00+00:00/2023-03-25T00:00:00+00:00", "segmentGranularity": "DAY"}
{"type": "compact", "dataSource": "my-datasource", "interval": "2023-02-18T00:00:00+00:00/2023-03-18T00:00:00+00:00", "segmentGranularity": "WEEK"}
"type": "compact", "dataSource": "my-datasource", "interval": "2022-01-24T00:00:00+00:00/2023-02-18T00:00:00+00:00", "segmentGranularity": "MONTH"}
Three different tasks
We do this once a week
v
oh wow those are some whacky compaction tasks. The 
interval
is not aligned to the 
segmentGranularity
in the 2nd and 3rd one. This should be an error, it is a shame that it isn't and we should change that.
also 
segmentGranularity: WEEK
is a poison (one you go WEEK you can never go MONTH or YEAR because weeks do not align with months and years)
In SQL based ingestion (which can also be used for compaction, replacing the 
compact
task) this would be an error
Just checked... in SQL based ingestion it would have given you this meaningful error:
so what is going on is: for your 3rd compaction (but your 2nd one is also sus): you select "2022-01-24T000000/2023-02-18T000000" and ask for month segments which will produce segments for "2022-01-01T000000/2023-02-01T000000" and "2022-02-01T000000/2023-03-01T000000" but each containing only part of a month so the segment for "2022-01-01T000000/2023-02-01T000000" would only contain 7 days of data but it will overshadow all of its time range so the data for "2022-01-01T000000/2022-01-24T000000" would be over shadowed and thus un-queryable and will appeal "lost". If you drop the new (crappy) month segment and mark all the segments it overshadowed as 
used
then the data be queryable again
we should make those 
compact
tasks throw an error to prevent confusion
But this leaves the question open: what are you actually trying to do? What compaction tasks will you submit now that you know that the ones you are submitting are errors (and will be marked as such in the future)
j
I am not sure if my issue is related to compaction tho. In my case it only happens with one datasource and in case it helps, my compaction task looks like:
Copy code
{
  "type": "compact",
  "id": "coordinator-issued_compact_geo_daily_cdmdddbk_2023-03-28T16:30:46.313Z",
  "resource": {
    "availabilityGroup": "coordinator-issued_compact_geo_daily_cdmdddbk_2023-03-28T16:30:46.313Z",
    "requiredCapacity": 1
  },
  "dataSource": "geo_daily",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2023-02-01T00:00:00.000Z/2023-03-01T00:00:00.000Z",
      "sha256OfSortedSegmentIds": null
    },
    "dropExisting": false
  },
  "dimensionsSpec": null,
  "transformSpec": null,
  "metricsSpec": null,
  "segmentGranularity": "MONTH",
  "granularitySpec": {
    "segmentGranularity": "MONTH",
    "queryGranularity": null,
    "rollup": null
  },
  ...
}
however I have to say that I have re-indexed “missed” data yesterday so, eventually data will be lost again since that has happened 3 times already while I try to find the root cause.
k
@Vadim Thanks! I didn't realize it behaved this way. That makes sense. I guess our goal here is to reduce the granularity of segments in the past in a rolling window. I.e. segments in the last week should be day-long. For the N weeks prior to that, week long, for the M months prior to that month long, at which point we drop them from the index. The rough purpose behind this approach is to make it easier to load data which is more likely to be loaded -- i.e. near-term data, while reducing the number of historical segments because we found that a large number placed a lot of load on the system. Is there a better approach to this? The existing auto compaction we run does hourly. I don't think it can scale the window into the past, though?
j
In my case I have found 50354 entries in 
druid_segments
with 
used = false
and the location to deep storage, so it looks like kill tasks are killing it. We do not run kill tasks tho
k
I think if the segment is still in the database it hasn't actually be killed, right? In my experience if it has been killed, nothing references it, and it is gone from the deepstore entirely.
j
k
Right. So, that's a good thing. It means your data is still there; it has just been dropped for some reason. 🙂 In my case we've dropped then killed the data, which means it's gone forever.
j
ah all right, so, dropped but not sure why
k
I just poked your thread with some of the queries I did to narrow down on the sort of data I was losing.
🙌 1
@Vadim we aren't using sql ingestion. Do you have a suggestion for a simple way to validate that the intervals I'm using are valid? l've rewritten our compaction task generator to drop the weekly compaction, and instead do only daily and monthly intervals. For the months, I've made it go from the 1st of the starting month at 00:00 to the 1st of the ending month at 00:00. I then start the daily compaction on the first of the ending month at 00:00. I'm assuming that the monthly compacting will take everything from the beginning of the 'start' month to the end of the month prior to the 'end' month, and the the daily one will do the same on a daily basis starting on the first of the 'end' month. But, I'm now a bit paranoid about the meaning of the monthly intervals.
v
I think you got it! For month-aligned intervals just validate that they both start and end have 
T00:00:00
in them!
You could just as well use a regex to validate them
/T00:00:00.+T00:00:00/
You should forget that WEEK exists. When do weeks even start? You could work it out but you definitely could not validate that with a regex
k
Awesome, thanks. WEEK? What WEEK? 🙂
v
Wait sorry, I forgot how time works. That I said above is for day-aligned intervals.
month-aligned would be /01T000000.+01T000000/
d
I second the opinion of never using WEEK. HOUR/DAY/MONTH only.
☝️ 1
g
PR to make non-aligned compaction specs an error (on the grounds that it's usually not what you want): https://github.com/apache/druid/pull/14127
5 Views
Open in Slack
PreviousNext
Powered by