https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
l
how are you deploying druid? did you set these CPU and memory values to it? what’s in the logs?
v
For now, we are manually configuring the druid cluster m6gd.large : https://instances.vantage.sh/aws/ec2/m6gd.large c6gd.4xlarge : https://instances.vantage.sh/aws/ec2/c6gd.4xlarge r6gd.4xlarge : https://instances.vantage.sh/aws/ec2/r6gd.4xlarge In logs
Copy code
T10:40:59,258 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: test_data_14b_2021-07-01T00:00:00.000Z_2021-08-01T00:00:00.000Z_2023-11-02T15:20:10.563Z_395, test_data_14b_2021-04-01T00:00:00.000Z_2021-05-01T00:00:00.000Z_2023-11-02T15:20:10.743Z_395, test_data_14b_2020-11-01T00:00:00.000Z_2020-12-01T00:00:00.000Z_2023-11-02T15:20:10.906Z_395, test_data_14b_2021-08-01T00:00:00.000Z_2021-09-01T00:00:00.000Z_2023-11-02T15:20:10.467Z_395, test_data_14b_2022-06-01T00:00:00.000Z_2022-07-01T00:00:00.000Z_2023-11-02T15:20:10.925Z_395, test_data_14b_2023-09-01T00:00:00.000Z_2023-10-01T00:00:00.000Z_2023-11-02T15:20:11.117Z_395, test_data_14b_2020-10-01T00:00:00.000Z_2020-11-01T00:00:00.000Z_2023-11-02T15:20:10.513Z_395, test_data_14b_2022-01-01T00:00:00.000Z_2022-02-01T00:00:00.000Z_2023-11-02T15:20:10.498Z_395, test_data_14b_2023-07-01T00:00:00.000Z_2023-08-01T00:00:00.000Z_2023-11-02T15:20:10.403Z_395, test_data_14b_2021-11-01T00:00:00.000Z_2021-12-01T00:00:00.000Z_2023-11-02T15:20:10.893Z_395, test_data_14b_2020-01-01T00:00:00.000Z_2020-02-01T00:00:00.000Z_2023-11-02T15:20:10.687Z_395, test_data_14b_2022-08-01T00:00:00.000Z_2022-09-01T00:00:00.000Z_2023-11-02T15:20:10.961Z_395, test_data_14b_2021-10-01T00:00:00.000Z_2021-11-01T00:00:00.000Z_2023-11-02T15:20:10.878Z_395, test_data_14b_2020-12-01T00:00:00.000Z_2021-01-01T00:00:00.000Z_2023-11-02T15:20:10.210Z_395, test_data_14b_2023-01-01T00:00:00.000Z_2023-02-01T00:00:00.000Z_2023-11-02T15:20:10.629Z_395, test_data_14b_2020-05-01T00:00:00.000Z_2020-06-01T00:00:00.000Z_2023-11-02T15:20:10.729Z_395, test_data_14b_2020-02-01T00:00:00.000Z_2020-03-01T00:00:00.000Z_2023-11-02T15:20:10.867Z_395, test_data_14b_2020-08-01T00:00:00.000Z_2020-09-01T00:00:00.000Z_2023-11-02T15:20:10.451Z_395, test_data_14b_2022-11-01T00:00:00.000Z_2022-12-01T00:00:00.000Z_2023-11-02T15:20:10.716Z_395, test_data_14b_2022-02-01T00:00:00.000Z_2022-03-01T00:00:00.000Z_2023-11-02T15:20:10.771Z_395, test_data_14b_2020-04-01T00:00:00.000Z_2020-05-01T00:00:00.000Z_2023-11-02T15:20:10.990Z_395, test_data_14b_2023-04-01T00:00:00.000Z_2023-05-01T00:00:00.000Z_2023-11-02T15:20:10.802Z_395, test_data_14b_2021-12-01T00:00:00.000Z_2022-01-01T00:00:00.000Z_2023-11-02T15:20:10.600Z_395, test_data_14b_2022-09-01T00:00:00.000Z_2022-10-01T00:00:00.000Z_2023-11-02T15:20:10.433Z_395, test_data_14b_2020-06-01T00:00:00.000Z_2020-07-01T00:00:00.000Z_2023-11-02T15:20:10.322Z_395, test_data_14b_2023-03-01T00:00:00.000Z_2023-04-01T00:00:00.000Z_2023-11-02T15:20:10.973Z_395, test_data_14b_2022-05-01T00:00:00.000Z_2022-06-01T00:00:00.000Z_2023-11-02T15:20:10.547Z_395, test_data_14b_2021-06-01T00:00:00.000Z_2021-07-01T00:00:00.000Z_2023-11-02T15:20:10.673Z_395, test_data_14b_2021-02-01T00:00:00.000Z_2021-03-01T00:00:00.000Z_2023-11-02T15:20:10.840Z_395, test_data_14b_2021-01-01T00:00:00.000Z_2021-02-01T00:00:00.000Z_2023-11-02T15:20:10.482Z_395, test_data_14b_2023-08-01T00:00:00.000Z_2023-09-01T00:00:00.000Z_2023-11-02T15:20:10.614Z_395, test_data_14b_2022-07-01T00:00:00.000Z_2022-08-01T00:00:00.000Z_2023-11-02T15:20:10.529Z_395, test_data_14b_2021-09-01T00:00:00.000Z_2021-10-01T00:00:00.000Z_2023-11-02T15:20:10.815Z_395, test_data_14b_2022-12-01T00:00:00.000Z_2023-01-01T00:00:00.000Z_2023-11-02T15:20:10.939Z_395, test_data_14b_2022-04-01T00:00:00.000Z_2022-05-01T00:00:00.000Z_2023-11-02T15:20:10.365Z_395, test_data_14b_2020-03-01T00:00:00.000Z_2020-04-01T00:00:00.000Z_2023-11-02T15:20:10.644Z_395, test_data_14b_2022-03-01T00:00:00.000Z_2022-04-01T00:00:00.000Z_2023-11-02T15:20:10.659Z_395, test_data_14b_2023-02-01T00:00:00.000Z_2023-03-01T00:00:00.000Z_2023-11-02T15:20:10.701Z_395, test_data_14b_2022-10-01T00:00:00.000Z_2022-11-01T00:00:00.000Z_2023-11-02T15:20:10.827Z_395, test_data_14b_2020-09-01T00:00:00.000Z_2020-10-01T00:00:00.000Z_2023-11-02T15:20:10.854Z_395, test_data_14b_2021-03-01T00:00:00.000Z_2021-04-01T00:00:00.000Z_2023-11-02T15:20:10.583Z_395, test_data_14b_2021-05-01T00:00:00.000Z_2021-06-01T00:00:00.000Z_2023-11-02T15:20:10.346Z_395, test_data_14b_2023-06-01T00:00:00.000Z_2023-07-01T00:00:00.000Z_2023-11-02T15:20:10.383Z_395, test_data_14b_2020-07-01T00:00:00.000Z_2020-08-01T00:00:00.000Z_2023-11-02T15:20:10.757Z_395, test_data_14b_2023-05-01T00:00:00.000Z_2023-06-01T00:00:00.000Z_2023-11-02T15:20:10.785Z_395
2023-11-07T10:40:59,258 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted stats: processed rows: [7902410900], persisted rows[625391], sinks: [45], total fireHydrants (across sinks): [135], persisted fireHydrants (across sinks): [45]
2023-11-07T10:41:00,516 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushing in-memory data to disk because (estimated) bytesCurrentlyInMemory[686031044] is greater than maxBytesInMemory[686030848].
2023-11-07T10:41:00,517 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted rows[625,312] and (estimated) bytes[207,121,468]
2023-11-07T10:41:00,777 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2021-07-01T00:00:00.000Z_2021-08-01T00:00:00.000Z_2023-11-02T15:20:10.563Z_395] spill[3] to disk in [259] ms (14,884 rows).
2023-11-07T10:41:00,951 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2021-04-01T00:00:00.000Z_2021-05-01T00:00:00.000Z_2023-11-02T15:20:10.743Z_395] spill[3] to disk in [173] ms (15,123 rows).
2023-11-07T10:41:01,102 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2020-11-01T00:00:00.000Z_2020-12-01T00:00:00.000Z_2023-11-02T15:20:10.906Z_395] spill[3] to disk in [150] ms (13,350 rows).
2023-11-07T10:41:01,280 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2021-08-01T00:00:00.000Z_2021-09-01T00:00:00.000Z_2023-11-02T15:20:10.467Z_395] spill[3] to disk in [177] ms (15,544 rows).
2023-11-07T10:41:01,447 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2022-06-01T00:00:00.000Z_2022-07-01T00:00:00.000Z_2023-11-02T15:20:10.925Z_395] spill[3] to disk in [166] ms (14,602 rows).
2023-11-07T10:41:01,454 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2023-09-01T00:00:00.000Z_2023-10-01T00:00:00.000Z_2023-11-02T15:20:11.117Z_395] spill[3] to disk in [6] ms (243 rows).
2023-11-07T10:41:01,627 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2020-10-01T00:00:00.000Z_2020-11-01T00:00:00.000Z_2023-11-02T15:20:10.513Z_395] spill[3] to disk in [172] ms (14,934 rows).
2023-11-07T10:41:01,803 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2022-01-01T00:00:00.000Z_2022-02-01T00:00:00.000Z_2023-11-02T15:20:10.498Z_395] spill[3] to disk in [175] ms (15,395 rows).
2023-11-07T10:41:02,037 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2023-07-01T00:00:00.000Z_2023-08-01T00:00:00.000Z_2023-11-02T15:20:10.403Z_395] spill[3] to disk in [233] ms (12,933 rows).
2023-11-07T10:41:02,214 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2021-11-01T00:00:00.000Z_2021-12-01T00:00:00.000Z_2023-11-02T15:20:10.893Z_395] spill[3] to disk in [176] ms (15,352 rows).
2023-11-07T10:41:02,399 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2020-01-01T00:00:00.000Z_2020-02-01T00:00:00.000Z_2023-11-02T15:20:10.687Z_395] spill[3] to disk in [184] ms (15,406 rows).
2023-11-07T10:41:02,579 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2022-08-01T00:00:00.000Z_2022-09-01T00:00:00.000Z_2023-11-02T15:20:10.961Z_395] spill[3] to disk in [178] ms (15,548 rows).
2023-11-07T10:41:02,762 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2021-10-01T00:00:00.000Z_2021-11-01T00:00:00.000Z_2023-11-02T15:20:10.878Z_395] spill[3] to disk in [182] ms (15,670 rows).
2023-11-07T10:41:02,909 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2020-12-01T00:00:00.000Z_2021-01-01T00:00:00.000Z_2023-11-02T15:20:10.210Z_395] spill[3] to disk in [146] ms (13,012 rows).
2023-11-07T10:41:03,089 INFO [[single_phase_sub_task_test_data_14b_ccmblgii_2023-11-05T04:21:00.546Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[test_data_14b_2023-01-01T00:00:00.000Z_2023-02-01T00:00:00.000Z_2023-11-02T15:20:10.629Z_395] spill[3] to disk in [179] ms (15,276 rows).
l
I understand the sizing of the nodes, but we need see the configuration. Like how many workers are set? Whats the Xmx and direct memory?
Your data nodes got 16 CPU, but you are limiting to 4 workers each. Usually safe to use half CPU as workers (so 8 in your case), even more if you understand your use case. Also I can see only 2/4 are in use in both MMs. Did you set a high value in max concurrent tasks in your ingestion? If it's your single (or most important) task, give it a value equals to the total number of workers in your cluster (so 8 now, 16 if you add more workers like I suggested). I still cant see JVM values you set.
v
Copy code
#
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements. See the NOTICE file
 # distributed with this work for additional information
 # regarding copyright ownership. The ASF licenses this file
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License. You may obtain a copy of the License at
 #
 #  <http://www.apache.org/licenses/LICENSE-2.0>
 #
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 # KIND, either express or implied. See the License for the
 # specific language governing permissions and limitations
 # under the License.
 #
 
druid.service=druid/middleManager
 druid.plaintextPort=8091
 
# Number of tasks per middleManager
 druid.worker.capacity=4
 
# Task launch parameters
 druid.indexer.runner.javaCommand=bin/run-java
 druid.indexer.runner.javaOptsArray=["-server","-Xms4g","-Xmx4g","-XX:MaxDirectMemorySize=4g","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]
 druid.indexer.task.baseTaskDir=var/druid/task
 
# HTTP server threads
 druid.server.http.numThreads=60
 druid.server.http.defaultQueryTimeout=60000
 
# Processing threads and buffers on Peons
 druid.indexer.fork.property.druid.processing.numMergeBuffers=2
 druid.indexer.fork.property.druid.processing.buffer.sizeBytes=500MiB
 druid.indexer.fork.property.druid.processing.numThreads=1
 
# Hadoop indexing
 druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
Copy code
druid.worker.capacity=4
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=500MiB
druid.indexer.fork.property.druid.processing.numThreads=1
Are you referring to this @Luiz Augusto
l
the two I was expecting to see are
Copy code
druid.worker.capacity=4

 druid.indexer.runner.javaOptsArray=["-server","-Xms4g","-Xmx4g","-XX:MaxDirectMemorySize=4g","-Duser.timezone=UTC","-Dfile.encoding=UTF-8","-XX:+ExitOnOutOfMemoryError","-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]
v
Okay
l
workers size are very good. Just bump worker capacity to 8. Then go to you ingestion spec and look for
Copy code
"maxNumConcurrentSubTasks": 4,
increase it 15. I’m assuming this 12 bi row dataset is composed by many S3 files, is that correct?
v
Nope it's a single CSV
l
Hmmm. I might be wrong, but I imagine this can force the first step to run over the file in a single task, and then the following steps can run in parallel.
v
Okay
k
Please share the task report of the msq controller
2 Views
Open in Slack
PreviousNext
Powered by