https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
j
Stacktrace :
Copy code
java.lang.RuntimeException: org.apache.druid.msq.indexing.error.MSQException: BroadcastTablesTooLarge: Size of the broadcast tables exceed the memory reserved for them (memory reserved for broadcast tables = 223640812 bytes)
 at org.apache.druid.java.util.common.Either.valueOrThrow(Either.java:91)
 at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.runProcessorNow(FrameProcessorExecutor.java:258)
 at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.run(FrameProcessorExecutor.java:137)
 at org.apache.druid.msq.exec.WorkerImpl$1$2.run(WorkerImpl.java:634)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:251)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.druid.msq.indexing.error.MSQException: BroadcastTablesTooLarge: Size of the broadcast tables exceed the memory reserved for them (memory reserved for broadcast tables = 223640812 bytes)
 at org.apache.druid.msq.querykit.BroadcastJoinHelper.buildBroadcastTablesIncrementally(BroadcastJoinHelper.java:120)
 at org.apache.druid.msq.querykit.BaseLeafFrameProcessor.initializeSegmentMapFn(BaseLeafFrameProcessor.java:155)
 at org.apache.druid.msq.querykit.BaseLeafFrameProcessor.runIncrementally(BaseLeafFrameProcessor.java:106)
 at org.apache.druid.frame.processor.FrameProcessors$1FrameProcessorWithBaggage.runIncrementally(FrameProcessors.java:70)
 at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.runProcessorNow(FrameProcessorExecutor.java:229)
 ... 8 more
Request :
Copy code
REPLACE INTO "table1"
OVERWRITE WHERE __time >= TIMESTAMP '2022-11-22 00:00:00' and __time < TIMESTAMP '2022-12-01 10:00:00'
WITH source as (
  SELECT *
  FROM "table1"
  WHERE __time >= TIMESTAMP '2022-11-22 00:00:00' and __time < TIMESTAMP '2022-12-01 10:00:00'
),
events as (
  SELECT
    *
  FROM "events"
  WHERE __time >= TIMESTAMP '2022-11-30 15:00:00'
)
SELECT
  source.__time,
  source.id,
  source.country,
  SUM(source."count") + SUM(events."count") as "count",
  GREATEST(MAX(source.click), MAX(events.click)) as "click",
  GREATEST(MAX(source.impression), MAX(events.impression)) as "impression"
FROM source
  LEFT JOIN events on events.id = source.id
GROUP BY 1, 2, 3
PARTITIONED BY HOUR
k
So it should be
Copy code
BigDataSource left join SmalDataSource on
You might not need all the cols from events. That will probably reduce the size
j
Hi Karan, thank you for your detailed answers! Unfortunately it’s already 
BigDataSource left join SmalDataSource on
, and I need every columns from events, so I will try increasing the memory of the peon, and run the query multiple time on smaller intervals for my events table.
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by