https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
b
Check for the retention rule in the Datasource page for the specific datasource.
a
ok
I have cluster default rule only one : loadforever rule with replicant set to 1 (as single machine) and tier set to _default_tier
b
Then retention should not have deleted the rows
check for the coordinator logs, and historical for any clue on the deleted segments
v
what is the task duration of the kafka task? If the task failed for somereason before committing the records then you will see this
a
Hi vijay, where I can see the task duration ? And for long hours yesterday till go home I could see the records in datasource, it mean it was committed right ? or still in not yet commit stage ?
But no loss in Kafka topic
v
the records will show up in the datasource though they are still in the ingestion task memeory. So if the task fails before writing segments to deep storage then a new task has to start from the earlier kafka offset
a
oh... ok
v
you can see task duration in ingestion spec "taskDuration": "PT3600S"
this default. 1 hour
a
so I guess the slot? worker capacity will be be trouble ?
since I have less than my data-sources?
and the slot / task would shared to others? I see PENDING state by today ? to that source
mostly duration was 10006 shows under Tasks in web console
v
worker capacity may be a problem as you need atleast two slots to run 1 ingestion task. The task will shut down in hour and handover to another but both tasks will run together for some time when the handover happens
a
"durationSeconds": 3600, yes
v
how many slots do you have and how many tasks?
a
7 I have (8 cpu - 1)
tasks (i.e. data-sources) 9
sorry 8 tasks
but data-sources 10
so within limited worker capacity, but having task duration set higher can manage ?
or must to increase slots / worker capacity ?
Vijay, In another project I have less tasks more worker capacity where I not find this issue...
But I like to understand and learn the reason.
v
how many worker slots do you have for running 8 tasks? You should have 10-12 slots so that atleast some slots are available for task cycling
a
okay, yes currently I have only 7 worker capacity (based on recommendation number of cpu - 1)
But 8 tasks ( 10 datasources)
g
generally if data shows up and then mysteriously disappears, it's often a problem with handoff
often these are issues with metadata store or deep storage configuration, so i would double-check that stuff
you want to make sure deep storage is shared between all servers and metadata store is shared between all possible leading masters (Coordinator/Overlord)
if you just have one server this is probably not the problem — so in that case i'd suggest looking at the ingest task logs to see if there are errors, or looking at 
sys.segments
table to see if you have any unavailable segments
that could indicate a problem getting them loaded up.
a
Hi Gian, Thank you for you details... Sorry delay msg as am not well for a week. Well Gian, 1. I tried (in mean time) by increasing the worker capacity and it's working till now. (to ensure all the tasks / data-source have enough slots to process and commits) 2. But as per recommendation of druid, worker capacity should => (no., of cpu - 1) 3. Don't know what other would caused disappears !
Open in Slack
PreviousNext
Powered by