https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
b
Copy code
13:50:49.695 [main-EventThread] ERROR org.apache.curator.ConnectionState - Authentication failed
How are you connecting to your broker?
s
Umm, not sure what you mean to ask, but i don't think there's any connectivity issue, as there's another datasource in the same druid setup which is ingesting data from the same kafka topic and the ingestion/querying is working properly for that datasource.
v
are you ingesting from the same topic into two different sources?
s
Yes
In fact, many, not just two
v
please paste the ingestion spec of this source that is failing and the other source from the same topic that is succeeding
Copy code
ISE: Could not allocate segment for row with timestamp[2023-03-27T13:50:43.548Z]
this usually means a metadata issue or a lock conflict.
s
Yeah, I read this one, but in my case, there isn't any existing segment which can be compacted, as this is a new datasource
v
please paste the spec
you can try changing the spec with a new data source different from the one you are trying to ingest. Perhaps this data source name was used earlier and there is some metadata?
s
tried that as well, didn't work, the new datasource went into RUNNING state for a couple of seconds and then went to UNHEALTHY_TASKS state again, and the tasks were failing with the same error that I have posted
Will paste the spec too after a while, right now I don't have access to the machine which has the ingestion spec
v
only this specific data source is failing?
s
yes, the other datasource which is getting data from the same topic is working fine
g
hmm. I've never seen errors like 
Could not allocate segment for row
on a fresh datasource, only on one with pre-existing data (when the pre-existing segment partitioning or 
segmentGranularity
is somehow incompatible with newly added realtime segments)
I wonder if you have a 
segmentGranularity
that doesn't support appends? It would need to be one of the standard ones like 
hour
or 
day
for realtime kafka ingest. You may have issues with a 
segmentGranularity
with a custom time zone or a period that isn't one of the standard ones
s
segmentGranularity is SIX_HOUR
found this in the logs:
Copy code
2023-03-27T13:50:05,730 ERROR [KafkaSupervisor-reporting_shard0-Worker-1] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Problem while getting checkpoints for task [index_kafka_reporting_shard0_31f5073b24d52a9_ejkmgick], killing the task
java.util.concurrent.ExecutionException: org.apache.druid.indexing.common.IndexTaskClient$TaskNotRunnableException: Aborting request because task [index_kafka_reporting_shard0_31f5073b24d52a9_ejkmgick] is not runnable
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_342]
	at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_342]
	at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.verifyAndMergeCheckpoints(SeekableStreamSupervisor.java:1581) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$verifyAndMergeCheckpoints$16(SeekableStreamSupervisor.java:1538) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_342]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_342]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_342]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_342]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: org.apache.druid.indexing.common.IndexTaskClient$TaskNotRunnableException: Aborting request because task [index_kafka_reporting_shard0_31f5073b24d52a9_ejkmgick] is not runnable
	at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:322) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.IndexTaskClient.submitRequestWithEmptyContent(IndexTaskClient.java:220) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClient.getCheckpoints(SeekableStreamIndexTaskClient.java:253) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskClient.lambda$getCheckpointsAsync$0(SeekableStreamIndexTaskClient.java:276) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	... 4 more
2023-03-27T13:50:05,731 WARN [KafkaSupervisor-reporting_shard0-Worker-1] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Clearing task group [1] information as no valid tasks left the group
g
Hmm I'm not sure why you'd have problems with `SIX_HOUR`…
0.19 is also fairly old; I wonder if you could try the latest too. If that still doesn't work then that'd be interesting, especially if you have steps to reproduce the problem
s
@Gian Merlino thanks for pointing out this error has something to do with segmentGranularity I tried changing the segment granularity from SIX_HOUR to DAY and the error has magically disappeared
2 Views
Open in Slack
PreviousNext
Powered by