https://linen.dev logo
Join Slack
Powered by
Hi guys...I'm trying to obtain that all the rows i...
# general
r
Hi guys...I'm trying to obtain that all the rows inserted in a datasource belong to one segment and that segment is related to current day...so the objective is to have one segment per day...I tried to set segmentGranularity:day in granularitySpec section but seeing the datasource there are 2 segments (and not one segment) related to the 2 rows present in the datasource. What I'm wrong? Are there others params to set to achive the behaviour described? Thanks
b
If the data belongs to one day and came in one run of the job, then the job will create single segment, But if the data is of two days then their will be two segments. If the data is of single day but came in consecutive job runs, it might create a segment with new partition Segmentname_1. Could you please share the __time column value and the segment names here.
r
The data are passed to Druid using /druid/indexer/v1/task API from REST controller. In a day, based on the interaction of the customers with the webapp, is possible that many times the API /druid/indexer/v1/task is called (with different data).
b
Correct, so it should be different runs and you should have new partitions created for the same day segment for each run. You can run compaction/enable auto compaction with day granularity and that will merge the segments into one. Another parameter that will decide on the number of same segment gets created is number of rows in a segment which is maxRowsPerSegment
r
The timestampSpec is: "timestampSpec": { "column": "@timestamp", "format": auto", because with "yyyy-MM-dd HHmmss" when I run the task I obtain "
Copy code
2024-06-12T08:43:10,759 ERROR [task-runner-0-priority-0] org.apache.druid.segment.incremental.ParseExceptionHandler - Encountered parse exception
org.apache.druid.java.util.common.parsers.ParseException: Timestamp[2024-06-12T08:45:19Z] is unparseable
while with "format":"auto" the task is executed correctly but in the datasource the field appears in this way:
the segment name is this
b
Could you share the complete ingestion spec? Auto is ok i believe if you want to see the complete time, we shoudl use query granularity as none along with segment granularity as day
r
maxRowsPerSegment is setted to the default value:
b
If you have multiple segments for a single day and you believe this should present in one segment and the record count is not much then you should enable auto compaction/run manual compaction. So that all segment partitions will get merged and only one segment will get created for a day.
r
yes I have multiple segments for a single day, but I don't want that behavior
so Druid not is able to create a single segment and populate it at ingestion time? How can enable auto compaction?
b
Enable auto compaction and that should merge all the rows to a single segment, make sure the compaction does have segment granularity set to Day. https://druid.apache.org/docs/latest/data-management/automatic-compaction/
r
I tried to execute this API for the auto compaction...
b
I believe you need to provide the interval. Check the spec if it covers the interval
r
On the datasource there are the following retention rules: 
[
{
"period": "P1D",
"includeFuture": true,
"tieredReplicants": {
"_default_tier": 2
},
"useDefaultTierForNull": true,
"type": "loadByPeriod"
},
{
"type": "dropForever"
}
]
but into the datasource view I see this:
I don't understand if is possible/where specify the interval...I only understand that the default indexing period is 30 minutes
b
skpioffset P1d is default. You can add below: “skipOffsetFromLatest”: “PT0S”, while running the job or edit the compaction configuration and change :
For auto compaction interval is not required
r
The segment granularity specified on the datasource is DAY, maybe for this reason I obtaining "4 segments skipped"?
b
if you are submitting manual compaction then interval is required
in your default config for compaction below is set: “skipOffsetFromLatest”: “PT1D”, Hence it is skipping the latest day segments, You can edit it and set it to PT0S, so that all segments will get compacted.
r
Yeah I need to have an automatic compaction...
b
Double Click on below:
Edit as below:
Then autocompaction should run and all segments should get compacted.
r
ok now I see this, so I have to wait 30 minutes to see if the compaction works or not, write?
b
Yes I believe, or you can just restart coordinator, that should run the job on restart
r
In my case I'm running Druid on Docker container so I don't know if it's possible to restart coordinator only...I have to investigate...thanks a lot
b
👍
r
ok now I see this, finally! 🙂
do you know why in the _time column I don't see seconds, minutes etc?
b
Try to use Query granulrity to none.
Copy code
"dataSchema": {
      "granularitySpec": {
        "segmentGranularity": "day",
        "queryGranularity": "none",
        "rollup": false
      },
r
yes it works! thanks
👍 1
but it is possible to change the column name from _time to the original column name?
b
__time is mandatory to have
if you do not map the original column to __time, then you can have two different column
r
👍
I believe that the automatic compaction doesn't working correctly:
there are many segments yet
b
It should work. Can you please share the screenshot from segments tab showing the complete 
Segment ID
for datasource 
ebsn-ecs-log-test
Auto Compaction we would not have much control over , What i can think of trying is to run manual compaction and see how that behaves. Sample SPEC: Please modify as per your requirement and then submit the job to check how it behaves.
Copy code
{
  "type": "compact",
  "dataSource": "ebsn-ecs-log-test",
  "ioConfig": {
    "type": "compact",
    "inputSpec": {
      "type": "interval",
      "interval": "2015-01-01/2025-02-01"
    }
  },
  "granularitySpec": {
    "segmentGranularity": "DAY"
  }
}
r
@Bibek Sahoo do you mean this?
I've submitted this task:
b
Can you please share the compaction task log
r
And I see yet 2 segment (but the total rows are much more then 5000000 rows, so I expected to have 1 segment with 4270 rows)
b
I believe these are two different date segments: 1 segment for 
12th to 13th
and 1 segment for 
13th to 14th
Pushing segments: [ ebsn-ecs-log-test_*2024-06-13*T000000.000Z_*2024-06-14*T000000.000Z_2024-06-13T072610.303Z, ebsn-ecs-log-test_*2024-06-12*T000000.000Z_*2024-06-13*T000000.000Z_2024-06-13T072610.303Z ]
r
ok and for ebs-log-search-test why there is the value running
I don't see any task started on that datasource, and the compaction configuration is the same of ebsn-ecs-log-test
b
Because you have enabled autocompaction. If you want to run manual compaction(as auto compaction does not full fill your requirement). You may disable autocompaction now. by deleting the autocompaction config.
Click on that running and compaction config will open, click on the delete, which will delete the auto compaction. Whenever required, you can run manual compaction as you did now.
r
ok but I need to have an autocompaction on both the datasources (we are testing it), and not the manual compaction (maybe we will use it only if really necessary)...and I don't understand why the autocompaction on that datasource is still in running from yesterday and the segment number grow...do you know why I don't see any compact task on ebsn-log-search task?
b
That will show always as running, We do not have to worry about that. Whenever coordinator finds any eligible segments, then it will submit a new compaction task.
Once coordinator do not find any new segment and completes all the time intervals, then it will update the status to Fully Compacted.
r
I thinked that with so many segments the autocompaction would start, ok thanks
👍 1
instead of "2015-01-01/2025-02-01" as interval Is it possible specify current_time in the place of ".../2025-02-01"?
b
The interval, I have given some default interval, you can provide as per your requirement, for example, if you want to compact from 10th of June to 13th of june, then you use 
"2024-06-10T00:00:00/2024-06-13T00:00:00"
OR `"2024-06-10/2024-06-13"`(this will internally get converted to 
2024-06-10T00:00:00/2024-06-13T00:00:00
)
👍 1
r
I runned a manual compaction of ebsn-log-search-test and the task is failed (Error when starting up. Failing.)
task-log-compact_ebsn-log-search-test_hdbihebd_2024-06-13T08_33_54.694Z.log
b
The job is failed because the time chunk for which you are running, the task is not getting the lock. Snippet from your task lock:
Copy code
Caused by: org.apache.druid.java.util.common.ISE: Lock for interval [2024-06-12T00:00:00.000Z/2024-06-14T00:00:00.000Z] was revoked.
You can check overlord log to see more about the lock.
Make sure no other jobs are running with same time interval for the same datasource.
r
If I add this in the task spec: 
"taskContext": {
"useConcurrentLocks": *true*,
"taskLockType": "REPLACE"
}
Should I resolve the problem or not?
b
Never used this. Tssks with high priority usually gets the lock, if you do not want the job to fail and if you are ok with other jobs(the ingestion) to fail having this timechunk, then you can run the job with high priority set in the context.
r
Ok I understand...surely I don't want that the ingestion tasks fail...
But from Druid console is it possible to see overlord logs?
on the machine where is installed Druid there are all the services except overload service...
3 Views
Open in Slack
PreviousNext
Powered by