Apache Pinot #general

Dan Hill

03/09/2020, 7:36 PM

I have a hybrid table. When I try to query it with a basic query, I get

Copy code

ProcessingException(errorCode: 450,
message: InternalError: <http://java.io|java.io>.IOException: Failed: HTTP error code: 500
at org.apache.pinot.controller.api.resources.PinotQueryResource.sendPostRaw(PinotQueryResource.java: 284)
at org.apache.pinot.controller.api.resources.PinotQueryResource.sendRequestRaw(PinotQueryResource.java: 322)
at org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java: 204)
at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java: 131)

When I look at the log, it looks like it cannot find any segments.

Copy code

java.lang.NullPointerException: null
        at org.apache.pinot.broker.routing.instanceselector.BalancedInstanceSelector.select(BalancedInstanceSelector.java:43) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.broker.routing.instanceselector.BaseInstanceSelector.select(BaseInstanceSelector.java:168) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.broker.routing.instanceselector.BalancedInstanceSelector.select(BalancedInstanceSelector.java:34) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.broker.routing.RoutingManager$RoutingEntry.calculateSegmentToInstanceMap(RoutingManager.java:517) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.broker.routing.RoutingManager.getRoutingTable(RoutingManager.java:416) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:300) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.broker.api.resources.PinotClientRequest.processSqlQueryPost(PinotClientRequest.java:146) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]

Dan Hill

03/09/2020, 7:37 PM

I've tried doing a batch load (which worked fine for non-hybird) and kafka streaming (I verified that a consumer gets the events).

Dan Hill

03/09/2020, 7:37 PM

Where should I look for how to debug this issue?

Neha Pawar

03/09/2020, 7:46 PM

What are the details of your stream? What's the table config?

Neha Pawar

03/09/2020, 7:48 PM

btw if you're only doing it for sample purposes, the instructions on https://apache-pinot.gitbook.io/apache-pinot-cookbook/getting-started/pushing-your-streaming-data-to-pinot just got done today

Dan Hill

03/09/2020, 7:57 PM

Here are the offline and realtime table configs

events-realtime-table.json events-offline-table.json

Dan Hill

03/09/2020, 7:58 PM

I'm using the KafkaJSONMessageDecoder. I noticed the docs saying that json isn't supported for streaming.

Neha Pawar

03/09/2020, 8:05 PM

That's true only for the quick start. JSON isn't supported only for loading sample data into stream. We use a utility called

StreamAvroIntoKafka

. We don't have a JSON equivalent yet.

Neha Pawar

03/09/2020, 8:08 PM

does your stream have events, or are you loading sample data into the stream?

Xiang Fu

03/09/2020, 8:20 PM

StreamAvroIntoKafka

has a parameter called

-outputFormat

you can specify

json

avro

Neha Pawar

03/09/2020, 8:24 PM

that's for output right. the input to StreamAvroIntoKafka has to be avro i thought? @User

Xiang Fu

03/09/2020, 8:24 PM

correct

Neha Pawar

03/09/2020, 8:25 PM

so that's what i meant. there's no way to load a sample json/csv into kafka directly. it needs to be converted to avro first

Xiang Fu

03/09/2020, 8:28 PM

Xiang Fu

03/09/2020, 8:29 PM

if it’s plain text, you can use kafka console producer

Xiang Fu

03/09/2020, 8:30 PM

Copy code

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-json-topic < test.json

Xiang Fu

03/09/2020, 8:31 PM

It will produce one line into one kafka message, so one json object should be compact into one line

👍 1

Dan Hill

03/09/2020, 8:38 PM

Yea, that's what I did.

Dan Hill

03/09/2020, 8:38 PM

It looks like Pinot is not finding segments for my table.

Dan Hill

03/09/2020, 8:39 PM

The pinot quick start runner is now listing "Segment upload failed".

Neha Pawar

03/09/2020, 8:51 PM

i just reran the realtime quickstart. Used

kafka-console-producer

. Works fine for me. My hunch is you have an incorrect config. Are the kafka zk port and kafka broker port correct?

Dan Hill

03/09/2020, 9:02 PM

I copied them from the airlineStats config.

Dan Hill

03/09/2020, 9:02 PM

I posted the offline and realtime table configs above.

Dan Hill

03/09/2020, 9:06 PM

My logs are showing

Copy code

java.nio.file.NoSuchFileException: /var/folders/_h/7cl8qfcd0cb4mhw_mmzytpcr0000gn/T/PinotController/192.168.0.29_9000/fileUploadTemp/events__0__0__20200309T1931Z.b744e958-ae2e-44e8-93df-a0155263978d
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_242]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_242]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_242]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_242]
        at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) ~[?:1.8.0_242]
        at java.nio.file.Files.newOutputStream(Files.java:216) ~[?:1.8.0_242]
        at java.nio.file.Files.copy(Files.java:3016) ~[?:1.8.0_242]
        at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.extractSegmentFromFormToLocalTempFile(LLCSegmentCompletionHandlers.java:404) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommit(LLCSegmentCompletionHandlers.java:257) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_242]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_242]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]

Neha Pawar

03/09/2020, 9:11 PM

is the timestamp in you data older than 5 days? i wonder if retention manager removed the segments

Dan Hill

03/09/2020, 9:20 PM

No, it's from Sunday.

Dan Hill

03/09/2020, 9:21 PM

Even if I try running

./bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile /my/dir/batch-job-spec.yml

, the query still fails. My expected dir has the segments.

Dan Hill

03/09/2020, 9:23 PM

Does airlineStats have a bad

Copy code

"stream.kafka.hlc.zk.connect.string": "localhost:2191/kafka",
   "stream.kafka.zk.broker.url": "localhost:2191/kafka",
   "stream.kafka.broker.list": "localhost:19092",

Dan Hill

03/09/2020, 9:27 PM

Can I not reuse the same kafka cluster as started up during the quick start script?

Neha Pawar

03/09/2020, 9:27 PM

quick start uses zk

localhost:2123

, your table config has 2191. Did you setup a separate ZK?