https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
i
Try checking the overlord log for information on the task
n
Hey Ian , looks like coordinator try to call middle-manger via rest ap and failed .. from coordinator - application log java.net.ConnectException: Connection refused: /10.1.3.215:8091
i
coordinator or overlord?
n
this is the log of coordinator .., I dont have a seaprate deploymebnt for overload ..
the default .. both processes deploy on the same host ..right ?
i
yes, on a master server, but their should be two separate logs, are they running as separate java processes?
n
I dont know .. I just speen up a "coordinator" command on docker ..
result of this was -
same java process .. I am gussing ..
i
ok and the middlemanager is running, correct?
n
yes .. on a diffrent ec2 node as docker container ..
i
ok, it looks like it is not able to reach that node, something at the network level
n
coordinator is failing reach middle-mangager ?
i
yes, the coordinator/overlord process seems to not be able to reach it?
n
ok ..
i will try to ssh to the coordinator container and do telnet to middle-manger
Thanks!! i will keep you posted ..
task start tunning after open port 8091 .. it run for 30 seconds and failed .. Failed to run task with an exception. See middleManager or indexer logs for more details.
what is the indexer -- is it coordinator service ?
i
no, it would be the middlemanager node in this case. You can search that log as well as the overlord/coordinator log to pinpoint the failure reason
n
indexer = middle-manger ?
i
yes in this instance. There should be a log on that middlemanager
n
ok I will check this , thanks!
so task is submited to middle-manger .. but failed on middle manger end ..
if I understand you correctly
i
that is correct, it is submitted to middlemanager which spawns a peon java process to run the task.
n
i see .. .. I will keep you posted ..
i
there is a diagram of the flow located here -> https://druid.apache.org/docs/latest/design/indexing-service.html
n
Hey Ian succed to have middle manger working .. was issue on the exposed port on docker ..
👍 1
Hey Ian , I am facing https://support.imply.io/hc/en-us/articles/360060443474-Historical-will-not-start-due[…]h-there-is-enough-heap-available-and-ulimit-is-set-very-high this issue exactly .. , currently I tried to delete "unused" segments .. running single historical docker container on i3en.3xlarge ec2 node with this config
Copy code
- Name: AWS_REGION
  Value: us-east-1
- Name: druid_historical_cache_useCache
  Value: true
- Name: druid_historical_cache_populateCache
  Value: true
- Name: druid_segmentCache_lazyLoadOnStart
  Value: true
  # set to 6.5 TB = 7146825580544 bytes (it should be smaller then /opt/data total disk space )
- Name: druid_cache_sizeInBytes
  Value: 7146825580544
- Name: druid_server_maxSize
  Value: 7146825580544
- Name: druid_segmentCache_locations
  Value: '[{"path": "/opt/data", "maxSize": "7146825580544"}]'
- Name: druid_segmentCache_locationSelector.strategy
  Value: mostAvailableSize
- Name: DRUID_XMX
  Value: '28g'
- Name: DRUID_XMS
  Value: '28g'
- Name: DRUID_MAXDIRECTMEMORYSIZE
  Value: 65536m
- Name: druid_processing_buffer_sizeBytes
  Value: 536870912
- Name: druid_processing_tmpDir
  Value: '/opt/data/processing/tmp'
using 7T local disk size to hold segments
can I change something on config before try to change max_map_count ?
just want to share that in addtion to clean old segment .. I reduce XMX to 15g and reduce DRUID_MAXDIRECTMEMORYSIZE to 13g . this free some memory on OS level .. that historical can use , those 3 action solved the issue without changing max_map_count
✅ 1
2 Views
Open in Slack
PreviousNext
Powered by