https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
v
how many tasks are you running? Each task will create one segment for that time chunk
v
task for thaat table.? I have not manually run any task i just created the table 3 weeks ago and left it alone
v
How did you create?
Batch ingestion?
v
no, real time ingestion from kafka
v
ok...how many partitions in kafka and how many concurrent sub tasks in druid?
Also note that kafka task will create 1 segment every hour by default (the task will ingest for 1 hour and shutdown and new task will start)
This is to ensure that if task fails then data is committed and you need to go back only one hour
v
the kafka queue has 3 partitions but data is only on first partition
v
you can turn on auto compaction to compact all these into 1 file
v
okay, the auto compaction settings needs to be done or automatic settings will do that.?
v
you have to enable auto compaction. The default settings should be good for your sceanrio
v
okay. thanks a lot.
but if i want to change the compaction settings i can do that right.? i saw the current compaction setting had offset at 1D but while making this table i had set it to 6M ,in fine tuning
task duration was set to be 6M in tunings
v
offset is to ensure that the data ingested most recently is not compacted (the real time task will typically hold a write lock on those segments)
v
now it is showing 360S when
v
change the offset to PT1H and enable compaction with everything else as default
you set task duration to 6 months? This is not a good thing. If the task fails then all the data of the 6 months will have to re-ingested. Real time tasks need to commit periodically.....you can use intermediate handoff period if you don't want the task to cycle
v
I think I have not understood this task duration setting
my final aim in this table is that i want my druid table to create only one segment every 6 months, because the data is less, it will be around 400mb hardly after 6 months
v
task duration essentially ensures that at the end of the duration the task stores the kafka offset and shutsdown after committing the data it has ingested to deep storage and another task will start and pick from that offset
so if a task fails before committing to deep storage then we just to start only from the offset that was stored by the previous task
v
okay, so does this task duration setting affect segment size.?
v
yes...each task will write one segment so in you case if the data cam in over 16 hours then you will have 16 segments
so set task duration 1 hour and enable compaction with PT1H offset. this will get you what you want
v
okay, thanks ,but this will create a segment every hour right.?
i want to create a segment every 6 months
v
no it will compact based on segment granularity and max rows per segment
👍 1
so one segment for the granularity if the rows don't exceed max rows per segment
👍 1
v
I Have set segment granularity as one year because data will be less
and max rows as 12 Lakhs
I done the settings as you said, thanks, but it is not reducing my already created segments. how to reduce them to one segment.?
b
You turned on compaction with offset PT1H? And the data (__time) is over 1H old? Do you see any compaction task logs in the Services view of the console? Do they show success?
v
yes i did that, it is showing : Target: 'Default 5,000,000' under compaction column in my druid data sources
v
You need to look at the tasks and see if compaction tasks ran and succeeded
v
i applied it for 2 tables, it is showing compaction task for only one and that table is also showing 46 segments for only kbs of data
after the compaction
s
I think you may still be running into a locking situation. If segment granularity of the real-time ingestion is set to 1 year, that means that it will hold a lock on the whole year while the streaming ingestion is running. Compaction will never be able to obtain the lock, because even after 1 hour of duration, as soon as a new event arrives in the next hour, the 1 year lock will be placed again and streaming task always has priority over compaction. I haven't tried this, but I think you can use mixed segment granularity, streaming with 1 hour duration and segment granularity of 1 hour. say that at the edge you may get events from the previous hour in the current one, so that you'd have up to 2 hourly segments locked at any given time. You would then set compaction with segment granularity of 1 year and a "skip Offset from latest" of 2 hours to avoid the streaming locks. I think this will achieve what you want, any streaming segments created more than 2 hours ago should accumulate into the 1-year segments. You'll need to use the JSON form of configuring auto-compaction so that you can specify the change in segment granularity in the granularitySpec. Here's an example I'm testing. It isn't streaming, but it is changing from hourly segment granularity into yearly:
Copy code
"dataSource": "kttm_transformed",
  "skipOffsetFromLatest": "PT2H",
  "tuningConfig": {
    "partitionsSpec": {
      "type": "dynamic",
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null
    },
    "type": "index_parallel"
  },
  "granularitySpec": {
    "segmentGranularity": "YEAR",
    "queryGranularity": null,
    "rollup": null
  }
}
I'll come back with the result once the compaction job runs.
turns out that the auto-compaction did not pick up old hourly segments, I think that is because it looks at recent segments (not sure what the limit to recent is). I ran a manual compaction task covering the whole time interval and it worked:
Copy code
{
  "type": "compact",
  "dataSource": "kttm_transformed",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2000-01-01T00:00:00.000Z/2021-01-01T00:00:00.000Z",
    },
  },
  "segmentGranularity": "YEAR",
  "granularitySpec": {
    "segmentGranularity": "YEAR",
  }
}
But I still think you can do this with auto-compaction because it will capture recent hourly segments and merge them into the year segment.
If you give it a try, let us know how it goes.
b
Also, how big is the 1-year segment (if you have one)? Do you set inputSegmentSizeBytes at all? If so, it needs to be bigger than the total size of all the segments being compacted. If you don't, then what Sergio said. 🙂
v
thanks for the help, will try these and let you know
2 Views
Open in Slack
PreviousNext
Powered by