Vijay Narayanan
01/16/2023, 8:02 AMBharat Thakur
01/16/2023, 8:04 AMVijay Narayanan
01/16/2023, 8:05 AMVijay Narayanan
01/16/2023, 8:05 AMswetha
01/16/2023, 8:41 AM2023-01-16T08:38:21,728 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-hjelfmnc-1, groupId=kafka-supervisor-hjelfmnc] Fetch position FetchPosition{offset=144333390, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-1.kafka-headless.druid.svc.cluster.local:9092 (id: 1 rack: null)], epoch=0}} is out of range for partition druid-0, raising error to the application since no reset policy is configured
2023-01-16T08:38:21,728 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=144333390, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-1.kafka-headless.druid.svc.cluster.local:9092 (id: 1 rack: null)], epoch=0}} is out of range for partition druid-0]
2023-01-16T08:38:21,728 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms
2023-01-16T08:38:51,733 INFO [task-runner-0-priority-0] org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=consumer-kafka-supervisor-hjelfmnc-1, groupId=kafka-supervisor-hjelfmnc] Fetch position FetchPosition{offset=144333390, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-1.kafka-headless.druid.svc.cluster.local:9092 (id: 1 rack: null)], epoch=0}} is out of range for partition druid-0, raising error to the application since no reset policy is configured
2023-01-16T08:38:51,734 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message [Fetch position FetchPosition{offset=144333390, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka-1.kafka-headless.druid.svc.cluster.local:9092 (id: 1 rack: null)], epoch=0}} is out of range for partition druid-0]
2023-01-16T08:38:51,734 WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - Retrying in 30000ms
Bharat Thakur
01/16/2023, 8:47 AMVaibhav Malik
01/16/2023, 3:53 PMD K
01/17/2023, 4:57 AMAkhila Mukka
01/17/2023, 5:28 AMD K
01/17/2023, 7:13 AMD K
01/17/2023, 7:13 AMD K
01/17/2023, 7:13 AMD K
01/17/2023, 7:14 AMIgor Berman
01/17/2023, 8:43 AMdruid.coordinator.asOverlord.enabled=true
druid.indexer.runner.type=remote
here is some log extraction from coordinator(and overlord) log with my comments(the relevant parts will be attached in thread)
index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_fnoejlpn
finished successfully ->
which triggered stopping all tasks in group(another one is index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn
Stopping all tasks in taskGroup[5] because: [Task [index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_fnoejlpn] completed successfully, stopping tasks [index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn, index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_fnoejlpn]]"
and it seems that it also finished:
2023-01-15T11:36:01,679 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[<http://taz-comp00674.taboolasyndication.com:8091|taz-comp00674.taboolasyndication.com:8091>] completed task[index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn] with status[SUCCESS]
2023-01-15T11:36:01,679 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[<http://taz-comp00674.taboolasyndication.com:8091|taz-comp00674.taboolasyndication.com:8091>] wrote SUCCESS status for task [index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn] on [TaskLocation{host='<http://taz-comp00674.taboolasyndication.com|taz-comp00674.taboolasyndication.com>', port=8102, tlsPort=-1}]
2023-01-15T11:36:01,679 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received SUCCESS status for task: index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn
but then:
2023-01-15T11:36:01,680 INFO [IndexTaskClient-sp_campaigns_realtime_aggregation-3] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Task [index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn] failed to stop in a timely manner, killing task
and then there is some race between 2 threads(Curator-PathChildrenCache-1
and IndexTaskClient-sp_campaigns_realtime_aggregation-3
)
2023-01-15T11:36:01,684 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn to status: TaskStatus{id=index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn, status=SUCCESS, duration=3727754, errorMsg=null}
2023-01-15T11:36:01,685 INFO [IndexTaskClient-sp_campaigns_realtime_aggregation-3] org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn to status: TaskStatus{id=index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn, status=FAILED, duration=-1, errorMsg=Task [index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn] failed to stop in a ti...}
Looking at logs of the index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn
task it finished before all those
when coordinator printed
2023-01-15T11:34:00,802 INFO [IndexTaskClient-sp_campaigns_realtime_aggregation-3] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClient - Task [index_kafka_sp_campaigns_realtime_aggregation_90e0b11d6585a1a_mhbgoepn] paused successfully
in it’s log I see
2023-01-15T11:36:00,754 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task
D K
01/17/2023, 2:12 PMBharat Thakur
01/17/2023, 3:52 PMBharat Thakur
01/17/2023, 3:52 PMDavid Monical
01/17/2023, 9:10 PMnull
values are stripped from multivalue fields when ingesting json data. Couldn't find anything in the documentation about this behavior so I filed a ticket here. Figured I'd open up the discussion here as well, thanks so much!MarekM
01/18/2023, 10:33 AMMarekM
01/18/2023, 10:33 AMMarekM
01/18/2023, 10:34 AMTomer Ben David
01/18/2023, 4:08 PM{
"queryType": "groupBy",
"dataSource": "wikipedia",
"granularity": "all",
"dimensions": [],
"aggregations": [
{
"type": "count",
"name": "my_count_sketch",
"aggregator": {
"type": "thetaSketch",
"name": "mycount",
"fieldName": "added"
}
},
{
"type": "arrayOfDoublesSketch",
"name": "my_array_of_double_sketch",
"fieldName": "my_count_sketch"
},
{
"type": "longSum",
"name": "deleted_sum",
"fieldName": "deleted"
},
{
"type": "count",
"name": "total_count"
}
],
"postAggregations": [
{
"type": "arithmetic",
"name": "average_deleted",
"fn": "/",
"fields": [
{
"type": "fieldAccess",
"fieldName": "deleted_sum"
},
{
"type": "fieldAccess",
"fieldName": "total_count"
}
]
},
{
"type": "arrayOfDoublesSketchToVariances",
"name": "my_variance_sketch",
"field": "my_array_of_double_sketch"
}
],
"intervals": [
"2014-01-15T00:00:00.000/2023-08-15T02:00:00.000"
]
}
This fails with:
Error: Unknown exception
Missing type id when trying to resolve subtype of [simple type, class org.apache.druid.query.aggregation.PostAggregator]: missing type id property 'type' (for POJO property 'field') at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 677] (through reference chain: org.apache.druid.query.groupby.GroupByQuery["postAggregations"]->java.util.ArrayList[1]->org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchToVariancesPostAggregator["field"])
com.fasterxml.jackson.databind.exc.InvalidTypeIdException
If I remove the block:
{
"type": "arrayOfDoublesSketchToVariances",
"name": "my_variance_sketch",
"field": "my_array_of_double_sketch"
}
Then the query passes with results.
Anyone can assist me please in getting a working example variance tuple sketch query for the wikipedia dataset? thanks!Michael Taranov
01/19/2023, 12:51 PMShilpa Sivanesan
01/19/2023, 1:28 PMShilpa Sivanesan
01/19/2023, 1:30 PMVijay Narayanan
01/19/2023, 1:38 PMShilpa Sivanesan
01/19/2023, 4:23 PMShilpa Sivanesan
01/19/2023, 4:35 PMclickstream_2022-10-12T00:00:00.000Z_2022-10-13T00:00:00.000Z_2022-12-12T09:08:08.316Z_9
clickstream_2022-10-12T00:00:00.000Z_2022-10-13T00:00:00.000Z_2022-12-12T09:08:08.316Z_9
whereas few segments are replicated to equal number of cold nodes
if cold nodes are 10 then segment is replicated 10 times
clickstream_2022-01-12T00:00:00.000Z_2022-01-13T00:00:00.000Z_2022-03-14T08:15:18.060Z_8 ---- this segment is replicated to 10
Shilpa Sivanesan
01/19/2023, 4:46 PMShilpa Sivanesan
01/19/2023, 4:53 PM