https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
a
Hey Nir – can you share a little more context. What were you doing when you saw this error? And where did you spot it? Seeing the port 8100 suggests it is between the Overlord and Middle Manager task – is that right? If so, is there any more information in the Overlord log at this time? Would also check the middle manager log at the same timestamp. Aside from this error, is your cluster healthy or are you encountering problems?
n
Thanks Adam , I suspect it was an issue with security group on aws .. where port 8100 was not expose .. but still checking .. 8100 is the sub-task port (peon) and 8091 is middle-manager port .. ASFAIU , in case overload and middle manger containers not running on the same ec2 instance
a
Sounds plausible! The logs should offer more insight.
n
added port mapping on middle manger container also for port 8100 and this solved the issue
a
Nice one Nir. thanks for updating how you fixed it.
n
sorry for the bammer .. it not fixed yet .. probebly I did not understnad the issue ..
exception - is on a sub-task log (peon) running on 172.17.0.6:8100 parent task (middel-manager) is on - 172.17.0.3:8100 stack-trace I dont undersntad from the log if the middle manger initate the (middle manger --> peon) or the other way around (peon --> middle manger) or (overload --> middle manager) ?? task submited with - org.apache.druid.indexing.common.IndexTaskClient.checkConnection , method is the check the connection between which parties ..?
Copy code
2022-12-05T17:20:28,674 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module]
2022-12-05T17:21:02,634 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.IndexTaskClient - Retries exhausted for [<http://172.17.0.3:8100/druid/worker/v1/chat/index_parallel_campaign_analytics_hourly_s3_gay_ioiedjfe_2022-12-05T17%3A20%3A08.821Z/segment/allocate>], last exception:
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_252]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_252]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_252]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_252]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_252]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_252]
	at java.net.Socket.connect(Socket.java:556) ~[?:1.8.0_252]
	at java.net.Socket.<init>(Socket.java:452) ~[?:1.8.0_252]
	at java.net.Socket.<init>(Socket.java:229) ~[?:1.8.0_252]
	at org.apache.druid.indexing.common.IndexTaskClient.checkConnection(IndexTaskClient.java:209) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:348) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.IndexTaskClient.submitSmileRequest(IndexTaskClient.java:258) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTaskClient.allocateSegment(ParallelIndexSupervisorTaskClient.java:52) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.task.SupervisorTaskCoordinatingSegmentAllocator.allocate(SupervisorTaskCoordinatingSegmentAllocator.java:56) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.getSegment(BaseAppenderatorDriver.java:334) ~[druid-server-0.19.0.jar:0.19.0]
	at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.append(BaseAppenderatorDriver.java:402) ~[druid-server-0.19.0.jar:0.19.0]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.add(BatchAppenderatorDriver.java:114) ~[druid-server-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:381) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.runTask(SinglePhaseSubTask.java:212) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:124) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:421) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:393) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_252]
on parent task (middel-manager) is on - 172.17.0.3:8100 get this
Copy code
022-12-05T17:20:14,391 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner - Waiting for subTasks to be completed
2022-12-05T17:21:04,349 WARN [task-monitor-0] org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor - task[single_phase_sub_task_campaign_analytics_hourly_s3_gay_pahpoeip_2022-12-05T17:20:14.374Z] failed!
2022-12-05T17:21:04,349 ERROR [task-monitor-0] org.apache.druid.indexing.common.task.batch.parallel.TaskMonitor - spec[index_parallel_campaign_analytics_hourly_s3_gay_ioiedjfe_2022-12-05T17:20:08.821Z_1] failed after [1] tries
2022-12-05T17:21:04,352 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexPhaseRunner - Failed because of the failed sub
Thanks!!
task submited with - "maxNumConcurrentSubTasks": 2, (so it allows 2 sub-tasks for a single middlemanger request) when running with "maxNumConcurrentSubTasks": 1 - single peon , task succeded. I suspect issue that durid take the docker ip , and not the ec2 host ip .. which cause the http to fail ?
got this exception on a peon sub-task log I am not sure I understand which component initiate this call .. is it (overload --> middle manger(main process) ? is it (middle manger (main process) --> peon (sub-task) ? is it ( peon (sub-task) ---> to middle manger) ? because this sub-task failure .. middle-manger task is failed . the checkConnection method .. is check the connection between which parties ?
Copy code
[<http://172.17.0.3:8100/druid/worker/v1/chat/index_parallel_campaign_analytics_hourly_s3_gay_ioiedjfe_2022-12-05T17%3A20%3A08.821Z/segment/allocate>], last exception:
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_252]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_252]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_252]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_252]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_252]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_252]
	at java.net.Socket.connect(Socket.java:556) ~[?:1.8.0_252]
	at java.net.Socket.<init>(Socket.java:452) ~[?:1.8.0_252]
	at java.net.Socket.<init>(Socket.java:229) ~[?:1.8.0_252]
	at org.apache.druid.indexing.common.IndexTaskClient.checkConnection(IndexTaskClient.java:209) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:348) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at org.apache.druid.indexing.common.IndexTaskClient.submitSmileRequest(IndexTaskClient.java:258) ~[druid-indexing-service-0.19.0.jar:0.19.0]
	at
can it be the sub-task is initating the call (peon --> middleManger(main)) , failed to comunicate with the main middle-manger process and failed .. I running 4 middle-manger containers , each 2 containers running on a diffrent ec2 instance
g
that's a call from the subtask to the main task
you can tell it's to the main task because the base url 
/druid/worker/v1/chat/<taskId>/
has the main task ID in it (starts with 
index_parallel
)
👍 1
you probably saw that error in a subtask log?
the subtask that logged it is the one trying to contact the main task (at 172.17.0.3:8100)
n
Thanks @Gian Merlino!! , issue I am facing is when I turn maxNumConcurrentSubTasks= 2 on the task spec , I have a middle manger(main) container that run on ec2-A node , and the sub-task container running on ec2-B node . so sub-task from ec2-B is failing to comunicate with middle manger ec2-A , because druid.host value on both containers (main and sub-task )is the internal docker ips 172.17.x.x any suggestion how this can be sovled .. running main and sub-task not on the same host with docker .. ?
g
hmm, i'm not a docker expert but i think you would need to do something with the network to ensure the different containers can communicate
n
Hey , I had some progress try to set custom values on those properties ... 
druid.indexer.runner.startPort/druid.indexer.runner.endPort
druid.indexer.runner.ports
open my middel-manager /status/properties , and I did not see the values I set .. running on version -
Copy code
druid-0.19.0
can be that those properties are not honer on this version ?
what happen that even when I set close list of ports (runner.ports) , in runtime the sub-task try to call with 8100
Hey , I had some progress try to set custom values on those properties ... 
druid.indexer.runner.startPort/druid.indexer.runner.endPort
druid.indexer.runner.ports
open my middel-manager /status/properties , and I did not see the values I set .. running on version -
Copy code
druid-0.19.0
can be that those properties are not honer on this version ? how can I validate those properties has any impact for druid 0.19.0 ?
please ignore - I had mistake on the property name , did not replace "." with "_" for docker env properties are honer on 0.19.0
g
good to know!
2 Views
Open in Slack
PreviousNext
Powered by