https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
m
@Vijay Narayanan 4 tasks are running
v
how many files do you have in s3?
and what kind of files?
m
around 61 files, each file is around 150MB
record count is around 340M for 3 months
but I'm apply filter for 1 month only
I'm using HLLSketchBuild in metrics one
all files are parquet
@Vijay Narayanan
s
Hi @Mohit Garg, Can you tell us a bit about the size of your cluster? • Share your Middle Manager configuration. • How many middle managers do you have and what resources in terms of CPUs and Memory do they have available? • What is the worker.capacity set to • What are your settings for 
druid.indexer.runner.javaOptsArray
? In addition to all that, and as an alternate path which is usually better, are you on Apache Druid 24.0.x ? One of the great new features is SQL based batch ingestion, where you can use a SQL INSERT or REPLACE statement to ingest external data and it has better performance than the traditional index_parallel batch ingestion. Rollup becomes a SQL aggregation query such as:
Copy code
INSERT INTO "kttm_rollup"

WITH kttm_data AS (
SELECT * FROM TABLE(
  EXTERN(
    '{"type":"http","uris":["<https://static.imply.io/example-data/kttm-v2/kttm-v2-2019-08-25.json.gz>"]}',
    '{"type":"json"}',
    '[{"name":"timestamp","type":"string"},{"name":"agent_category","type":"string"},{"name":"agent_type","type":"string"},{"name":"browser","type":"string"},{"name":"browser_version","type":"string"},{"name":"city","type":"string"},{"name":"continent","type":"string"},{"name":"country","type":"string"},{"name":"version","type":"string"},{"name":"event_type","type":"string"},{"name":"event_subtype","type":"string"},{"name":"loaded_image","type":"string"},{"name":"adblock_list","type":"string"},{"name":"forwarded_for","type":"string"},{"name":"language","type":"string"},{"name":"number","type":"long"},{"name":"os","type":"string"},{"name":"path","type":"string"},{"name":"platform","type":"string"},{"name":"referrer","type":"string"},{"name":"referrer_host","type":"string"},{"name":"region","type":"string"},{"name":"remote_address","type":"string"},{"name":"screen","type":"string"},{"name":"session","type":"string"},{"name":"session_length","type":"long"},{"name":"timezone","type":"string"},{"name":"timezone_offset","type":"long"},{"name":"window","type":"string"}]'
  )
))

SELECT
  FLOOR(TIME_PARSE("timestamp") TO MINUTE) AS __time,
  session,
  agent_category,
  agent_type,
  browser,
  browser_version,
  MV_TO_ARRAY("language") AS "language", -- Multi-value string dimension
  os,
  city,
  country,
  forwarded_for AS ip_address,

  COUNT(*) AS "cnt",
  SUM(session_length) AS session_length,
  APPROX_COUNT_DISTINCT_DS_HLL(event_type) AS unique_event_types
FROM kttm_data
WHERE os = 'iOS'
GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
PARTITIONED BY HOUR
CLUSTERED BY browser, session
The EXTERN table function can access any batch source that traditional batch ingestion supports, so you can use 
"type":"s3"
and in the input format type you can use 
"type":"parquet"
. This should be directly transferable from you current ioConfig and inputFormat settings in your current batch ingestion.
@Mohit Garg any luck in figuring out what was making it slow?
m
Hi @Sergio Ferragut, Thanks for the suggestion, I need to try out the sql ingestion, I'll get back to you asap
@Sergio Ferragut @Vijay Narayanan Middle Manager Conf
Copy code
druid_node_type: 'middleManager'
  druid_worker_capacity: 10
  DRUID_XMX: 5G
  DRUID_XMS: 1G
  DRUID_MAXDIRECTMEMORYSIZE: 2g
  druid_processing_buffer_sizeBytes: '200000000'
  druid_processing_numMergeBuffers: 2
  druid_processing_numThreads: 2
  druid_server_http_numThreads: 250
  druid_indexer_runner_javaOptsArray: '["-server", "-Xms512m", "-Xmx1g", "-XX:MaxDirectMemorySize=2g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'
  druid_indexer_fork_property_druid_processing_buffer_sizeBytes: '200000000'
  druid_indexer_fork_property_druid_processing_numMergeBuffers: 2
  druid_indexer_fork_property_druid_processing_numThreads: 2
  druid_indexer_fork_property_druid_server_http_numThreads: 70
  #new changes
  druid_indexer_task_baseDir: "/opt/druid/var/druid/task_baseDir"
  druid_indexer_task_baseTaskDir: "/opt/druid/var/druid/baseTaskDir"
  druid_indexer_task_restoreTasksOnRestart: true
  druid_processing_tmpDir: "/opt/druid/var/druid/tmpDir"
  druid_indexer_storage_type: "metadata"
  druid_realtime_cache_useCache: true
  druid_realtime_cache_populateCache: true
  druid_cache_type: caffeine
  #groupby
  druid_query_groupBy_maxMergingDictionarySize: "100000000"
  druid_query_groupBy_maxOnDiskStorage: "10000000000"
s
Thanks @Mohit Garg. Just to confirm the fit, what resources in terms of CPU/Memory does each Middle Manager have? A few items caught my attention about the config: • the -xms and -xmx values are recommended to be set to the same value for both middle manager and peons (javaOptsArray). • The middle manager process itself does not need a lot of memory, so you can probably lower that to 1G or less and increase the heap for the peons which will help during ingestion reducing the need to spill to disk. • Think of splitting available memory among the peons, you have 10 peons configured - with 1G heap and 2G max direct memory. Take a look at the guidelines here which would indicate that max direct memory with you settings, would be more like 
(druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes
for the forked process = 1G, and more heap will help, so you could invert them heap=2G and maxDirect = 1G • that implies a total of 3G * 10 peons = 30G - if you have more, add more to the heap. • worker capacity set to 10 implies that you have 11 CPUs available to the MM. If you have less CPUs, reduce this number and redistribute the memory among the number of peons you end up with. • If you go down the SQL-Based Ingestion path, there's a really good explanation of how memory is used in the peons/workers here. Finally in a little shameless self promotion, I wrote this blog post yesterday speaking of using SQL to do Rollup ingestion, including approximations. I hope you find it useful.
4 Views
Open in Slack
PreviousNext
Powered by