https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
a
A few questions which may help us track it down... Can you confirm where you took the dataset from - maybe share the link. Also when you say this is a small instance of druid, I take it you mean you did the small quickstart? What version of Druid are you using? Did you accept default values for the quickstart & batch ingest or tune anything like in the ingestion spec or JVM? Seems like you've identified the 56th file as being the spot where there was a problem. Is it possible to ingest that file on it's own to prove it is valid? You could also review the Overlord log, task log and middle manager logs to see if there is more info.
c
Attached is an image showing which data I was fetching. I added it using the "connect external data" button on the "query" page. I'm pretty much running defaults for everything. I'm using 
apache-druid-24.0.1/bin/start-single-server-small
. I'll go search the other logs to see if there's anything notable and once I've copied those files then I'll try to run the 56th & 57th file to see if the issue reoccurs.
On the overseer logs I'm seeing a lot of 
2022-11-30T22:43:50,235 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - No available [_default_tier] servers or node capacity to assign segment [trips_xaa_2014-04-26T00:00:00.000Z_2014-04-27T00:00:00Z_2022-11-29T11:30:27.958Z]! Current replication: [[_default_tier:1/2]]
Getting more than 20 of those every millisecond. Makes it hard to find actual information
a
The files being loaded are of this format: https://static.imply.io/example-data/trips/trips_xcc.csv.gz https://static.imply.io/example-data/trips/trips_xcd.csv.gz etc. You can get the full list in the payload of the original job. So you should be able to redo the ingest but for a single file (query, connect external data) but now select HTTP for the location and paste one of these links in. Be interesting to also test loading this file using "batch classic" as well as MSQ. For the overseer logs, maybe you can make it more manageable by doing something like ... grep -v WARN overseerlogfile >> overseer.new.log. That should filter out the swarm of WARN messages. Then look for FAIL/ERROR/FATAL messages in this overseer.new.log.
🙌 1
g
guessing by the error you posted -- i think the connection to the http server that hosts the files got lost partway through a download
you could download them first and switch druid ingestion to use a 
local
input source
i'll try it too and see if it works for me
and report back in some number of hours 🙂
a
I tried loading these individually (the one that errored out and a couple each side) and they completed without error. So a timeout is most likely (or a resource issue)
b
FWIW, I tried the same configuration ( 
bin/start-single-server-small
) for the NYC cab data for MSQE and the data set is ingested fully with no errors. So I concur with what Adam and Gian has stated about 
http timeout
c
Yes when I tried those files individually then I found that there wasn't any issue with the files themselves. A timeout error matches up with an "unexpected end of stream" error, too. I'll make sure to anticipate such errors when designing the backend that's going to integrate with druid. Thank you all for the valuable insights!
🙌 2
Hmm.. tried twice to load the nyc cabs data from a local disc to a single server - first for the whole dataset and then only for the first 26 input chunks. Both attempts failed with the following OutOfMemory error after about 10 hours. I'm going to send off another batch now with just 13 input chunks but I feel there must be something I'm missing. I'm currently using the 
batch
index_parallel
import from local files, with the filter 
trips_xa*.csv.gz
. Every other setting is left at its default.
Copy code
2022-12-04T21:19:48,994 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1035) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:526) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:187) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1199) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:532) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:187) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:149) ~[druid-server-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:133) ~[druid-server-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:122) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:922) ~[druid-indexing-service-24.0.1.jar:24.0.1]
	... 11 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
	at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:102) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.CompressedPools$4.get(CompressedPools.java:95) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:170) ~[druid-core-24.0.1.jar:24.0.1]
	at org.apache.druid.collections.StupidPool.take(StupidPool.java:153) ~[druid-core-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.CompressedPools.getByteBuf(CompressedPools.java:110) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:49) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.DecompressingByteBufferObjectStrategy.fromByteBuffer(DecompressingByteBufferObjectStrategy.java:28) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.GenericIndexed$BufferIndexed.get(GenericIndexed.java:483) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.BlockLayoutColumnarDoublesSupplier$BlockLayoutColumnarDoubles.loadBuffer(BlockLayoutColumnarDoublesSupplier.java:173) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.BlockLayoutColumnarDoublesSupplier$1.get(BlockLayoutColumnarDoublesSupplier.java:68) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.data.ColumnarDoubles$1HistoricalDoubleColumnSelector.getDouble(ColumnarDoubles.java:79) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.selector.settable.SettableDoubleColumnValueSelector.setValueFrom(SettableDoubleColumnValueSelector.java:36) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.QueryableIndexIndexableAdapter$RowIteratorImpl.setRowPointerValues(QueryableIndexIndexableAdapter.java:322) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.QueryableIndexIndexableAdapter$RowIteratorImpl.moveToNext(QueryableIndexIndexableAdapter.java:301) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.ForwardingRowIterator.moveToNext(ForwardingRowIterator.java:62) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.MergingRowIterator.lambda$new$0(MergingRowIterator.java:84) ~[druid-processing-24.0.1.jar:24.0.1]
	at java.util.stream.IntPipeline$9$1.accept(IntPipeline.java:368) ~[?:?]
	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104) ~[?:?]
	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:699) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517) ~[?:?]
	at org.apache.druid.segment.MergingRowIterator.<init>(MergingRowIterator.java:92) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.makeMergedTimeAndDimsIterator(IndexMergerV9.java:1360) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:293) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:1303) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.multiphaseMerge(IndexMergerV9.java:1120) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:1062) ~[druid-processing-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:923) ~[druid-server-24.0.1.jar:24.0.1]
	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:784) ~[druid-server-24.0.1.jar:24.0.1]
	at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
	at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) ~[guava-16.0.1.jar:?]
	... 3 more
g
which start command are you using?
(like 
bin/start-single-server-large
or something like that)
& did you change the # of tasks or are you using the default (2)?
btw, on our end we just tried this dataset with 20 tasks (below), it took a little less than 2.5 hrs (reading from http). we also tried a smaller server with 2 tasks and ended up running out of disk space before hitting any of the errors you hit. we're trying again with a bigger disk, but in the meantime, i wonder if the problem you are having is linked to the particular combination of startup script and task count — that's why i asked. i'd like to get to the bottom of this, since we want the out of box setup to always yield good results 🧐
c
My disc is 500GB exactly. Druid gets pretty much all of that, besides the Debian install. The machine is a 
start-single-server-small
and fits the definition - 8vCPU, 64GB RAM
I tried using more than 2 tasks before but the database would hang and not do any work. From reading the documentation I took that to mean that you can only have one worker per physical (virtual) machine.
Since that OutOfMemory exception I posted, I've been gradually increasing load to see where the point of error is. Loading 8 chunks at once seems to be ok - I'm now trying 10. The point where it failed before was loading 13 chunks at once.
(Chunks meaning local 
.<http://csv.tz|csv.tz>
files)
g
hmm… you can definitely do more than one worker per physical machine. although the out of box config may be limited to 2 or 3, it's adjustable let me try the exact config you're using… 
bin/start-single-server-small
, with 2 tasks, on a machine with 64 GB RAM
when you see errors at various numbers of local files, is it always that 
OutOfMemoryError: Direct buffer memory
or do you see other ones too?
what SQL are you running? want to make sure i'm doing the same thing
c
Considering this is just a benchmarking vm, I should probably be able to generate an image for you if you'd be interested - I'd have to let this job finish first which might take 6 hours or might fail depending on our luck haha
I'm not sure what you're referring to regarding SQL - I don't remember installing any variant of SQL when setting up Druid and I've been using the batch ingestion since I started loading local files rather than remote ones.
a
I've also kicked this off. I took an r5.2xlarge EC2 instance (64g RAM) and started the quickstart with start-single-server-small. Just so we are on the same page, I followed these steps: Query -> Connect External Data -> Example Data -> NYC Cabs (All files) -> Load with defaults: Max Tasks 2. Current estimate this will take another 8 or 9 hours.
When you say you were using docker, are you building a custom image rather than using the druid/docker quickstart we provide (see here)?
c
I'm not actually using docker directly - I've been benchmarking / playing with Druid on GCP and so have options to export images and am using vCPUs etc
I used the conventional druid quickstart and initially used the 
Query -> Connect External Data
route, although since hitting that timeout error at the top of the thread, I've since been using the 
index_parallel
batch loading method instead. I have the ingestion spec to hand if you would find that convenient
g
dang, I was running this on my laptop overnight but it went to sleep!
i thought it might stay awake but nope
i'll try again in a VM
ah okay I was trying using SQL (which I guessed you were using initially due to the stack trace including "msq" in your initial error. that's associated with SQL based ingestion)
c
Aah no all that time lost! Yes even my VM has discarded the druid server at times when I've forgotten to disown the process before logging off. In other news, the 10-file ingestion operation succeeded, which means the failure point is somewhere between 11 and 13 inclusive.
I'm now going to see if I can generate an image of my disc and make that publicly available
g
let me back up a sec and say some stuff that may be clarifying 1) we have two ways of doing batch ingest (or three if you count hadoop, but let's ignore that, it's not common): native batch, and SQL. doc page describing the options is at https://druid.apache.org/docs/latest/ingestion/index.html#batch. In the web console you get SQL in the Query view 2) native batch is older and is associated with a bunch of usability and operational struggles: it requires tuning of memory, is slower, etc. SQL is newer and has an improved backend-- it was just released in the latest major Druid release, 24, with a goal of having ingest be much more usable and getting rid of the need to tune parameters to avoid OutOfMemoryErrors and the like. it does pretty well at this so far! but there are things we do run into here and there, some of which we're fixing in 25. we're very focused on making it rock solid and deprecating native batch as soon as we can. so i recommend using SQL for that reason 🙂. we're going to try to figure out why you got that http error and ensure there's some kind of appropriate handling of whatever happened. if you run into any issues with SQL ingest using local files we will hunt those down too. 3) parallelism for both options is dictated by the number of worker tasks you use. you can use more than one task per physical machine, up to the number of task slots you have. you can see the number in the Servers tab under middleManagers. I think for a "small" setup the default is 3. There's an issue here with the single server configs: you don't get the full prallelism of your machine! You have 8 vCPUs but can use at most 3 of them for ingestion. the idea here is that in a prod setup, you likely don't want to devote the full resources of your machine to ingest. you likely want the majority available for querying. but for an ingest benchmark, or if you're not doing active querying, this isn't ideal. this is an area we are considering some improvements in, likely an update to the single server configs to increase the number of slots. you can also do this manually using the runtime properties file. if you're interested, let me know and i can advise you on some properties to use
c
Goodness me you are too kind to give me such detailed and targeted help! I'm only expecting queries during working hours and I'm expecting batches of input data only at the end of the day after everybody goes home, so it'd be great if I could sacrifice some query availability for the sake of ingestion performance
Yes I had noticed that the classic batch system was discouraged, but was uncertain whether the new SQL batching supported local files
I realise now that I can reuse the native batch input source logic for the SQL queries. My mistake!
I've now launched an SQL ingestion that's reading 13 files from the local disc - we'll see whether or not I get any OutOfMemory issues - if not then we can probably put the whole issue to bed 🙂
g
ok, looking forward to it!
btw, one of us on our end did try it with a single worker on a small server using https, and ran into a similar gzip thing eventually. so it's not just you!
so, we're looking into why it happens
Goodness me you are too kind to give me such detailed and targeted help!
no problem, as you can tell we're very motivated to make the batch ingest experience rock solid — big part of that is people should not run into weird errors!
😃 1
I'm only expecting queries during working hours and I'm expecting batches of input data only at the end of the day after everybody goes home, so it'd be great if I could sacrifice some query availability for the sake of ingestion performance
yeah, for this, you'd definitely want to adjust the builtin configs. try setting 
druid.worker.capacity=9
(instead of default: 3) in 
conf/druid/single-server/small/middleManager/runtime.properties
, and setting 
maxNumTasks: 9
with your SQL ingest (or setting "Max tasks: 9" in the web console). that should get you maxed out on the CPUs you have
🙌 1
the main risk of raising 
druid.worker.capacity
is you might run out of memory if it's raised too high. but i think 9 would be safe on the 
small
config -- however if you run into memory issues after raising it, let us know since it may mean i did my math wrong 🙂
one thing we're working on for the near future is a script to do this math for you and spit out configs optimized for your hardware — @Rishabh has been looking into that in https://github.com/apache/druid/pull/13365. first version won't include an "ingestion optimized" mode but i am thinking this is a nice idea for a follow up
👀 2
a
I ran the 'NYC Taxi Cabs' example ingest overnight. It processed all 74 files without any memory errors (although I did eventually run out of disk space - doh!) I used the 'single-server/small' quickstart on aa 64g linux server. My point being we can definitely rule out any issues with the files themselves and consider this to be environmental/resource related. To summarise, you do not need to use SQL to ingest data, you can do native batch load (Load data -> Batch Classic). Hopefully this will get you moving!
🙌 1
c
Funnily enough my ingest of 13 files also failed because I ran out of disk space haha. I tried to do something clever by moving the local files into a storage bucket and linking it via FUSE but unfortunately I ran into an 
AccessDeniedException
. I also tried using a symlink to try and convince Java to delegate file access to bash, which can access the mounted bucket just fine - but alas Druid doesn't error but also can't see the files. I've compromised by keeping just those 13 files on my local disk and seeing if I can get them ingested. I've also cranked up the config to try and use all 8 cores - I'll report back with results 🙂
I've managed to ingest another 26 files worth of data but I'm out of storage space now - I think I'm happy with the reliability of this ingestion 🙂
Thank you all very much!
a
Good to hear. Best of luck with the rest of your work. we are here if you need us!
😄 1
2 Views
Open in Slack
PreviousNext
Powered by