https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • p
    @User Thanks a lot for enabling the TEXT_MATCH feature on dictionary encoded columns. on a smaller table with ~25M rows, simple regexp_like query takes 178ms vs TEXT_MATCH takes ~30ms This is pretty cool.
    🎉 3
    👍 2
  • r
    Thanks for sharing 🙂
    👍 2
  • b
    Hey everyone, what does this usually indicate? Did the query fail in broker itself or did it fail in the servers? Not quite clear
  • b
    Copy code
    org.apache.pinot.client.PinotClientException: Query had processing exceptions:
[{"errorCode":200,"message":"QueryExecutionError:\njava.lang.RuntimeException: Caught exception while running CombinePlanNode.\n\tat org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:148)\n\tat org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:38)\n\tat org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:220)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:152)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:136)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)\n\tat shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)"}]
	at org.apache.pinot.client.Connection.execute(Connection.java:129) ~[pinot-java-client-0.3.0.jar:0.3.0-9b2dc20c07dec6cf33df08c4444d996e8202c3ba]
	at org.apache.pinot.client.Connection.execute(Connection.java:96) ~[pinot-java-client-0.3.0.jar:0.3.0-9b2dc20c07dec6cf33df08c4444d996e8202c3ba]
	at org.apache.pinot.client.PreparedStatement.execute(PreparedStatement.java:72) ~[pinot-java-client-0.3.0.jar:0.3.0-9b2dc20c07dec6cf33df08c4444d996e8202c3ba]
	at org.hypertrace.core.query.service.pinot.PinotClientFactory$PinotClient.executeQuery(PinotClientFactory.java:82) ~[query-service-impl-0.1.1.jar:?]
	at org.hypertrace.core.query.service.pinot.PinotBasedRequestHandler.handleRequest(PinotBasedRequestHandler.java:113) ~[query-service-impl-0.1.1.jar:?]
	at org.hypertrace.core.query.service.QueryServiceImpl.execute(QueryServiceImpl.java:99) [query-service-impl-0.1.1.jar:?]
	at org.hypertrace.core.query.service.api.QueryServiceGrpc$MethodHandlers.invoke(QueryServiceGrpc.java:210) [query-service-api-0.1.1.jar:?]
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172) [grpc-stub-1.30.2.jar:1.30.2]
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) [grpc-api-1.30.2.jar:1.30.2]
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) [grpc-api-1.30.2.jar:1.30.2]
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40) [grpc-api-1.30.2.jar:1.30.2]
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86) [grpc-api-1.30.2.jar:1.30.2]
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) [grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:820) [grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.30.2.jar:1.30.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
  • m
    The exception trace is from the server
  • b
    What could be the cause of this?
  • k
    Coming soon - New cluster manager UI for Pinot! https://github.com/apache/incubator-pinot/pull/5732 Feedback welcome! Anyone with UX/UI experience who can help us make it better, please ping us 🙂. The first version is readonly. This was one of the most requested feature in recent polls https://poll.ly/#/2YeaKQ5o
    🎉 13
  • s
    Hi all, we are currently developing segment merge service on Pinot and the design document is available for general. Please take a look and provide us the feedback 🙂 https://docs.google.com/document/d/1-AKCfXNXdoNjFIvJ87wjWwFM_38gS0NCwFrIYjYsqp8/edit?usp=sharing
    ➕ 5
  • h
    Hey Guys! I'm honored to announced that now is available to use Apache Pinot with PYTHON SQL DB-API! please check this out: https://github.com/blcksrx/pypinot
    ✌️ 3
    ➕ 6
  • h
    @User @User Updated the Kafka Sink connector and added more configurations for batch and streaming. I also added initial code for generating the pinot segment. The segment generator is boilerplates at this moment. I would appreciate your guidance there. I didn’t want to go to far without reviewing it with you and ensure we are on the right path. The repo is @ https://github.com/mhomaid/kafka-connect-apache-pinot/
  • k
    Thanks @User Will take a look at it. Let’s create an issue and start discussion?
  • h
    Great
  • k
    It’s a key value pairs
  • k
    @User will be presenting Pinot at Kafka Summit. https://twitter.com/nehapawar18/status/1286827588208308224?s=21 aug 24 1pm pst. This years Kafka Summit virtual and free.
    🎉 12
  • a
    hi! what is the best way to make Pinot multiregional? Is it possible to do without consuming from kafka (rely on DFS replication of uploaded segments) to save $$$ ?
  • k
    yes, you can push segments from one region to other by using the upload call
  • k
    but this is only applicable for the flushed segments
  • m
    We only have upload capability for offline table (part of hybrid table) though, right?
  • m
    @User Do you have a hybrid or realtime-only table?
  • a
    mostly hypothetical question. Observability team had a thought experiment - can they use Pinot instead of ES to store logs. But they want geo redundancy
  • a
    so, tables will be whatever it needs to be 🙂
  • m
    Ok, the way we do this at LinkedIn is via Kafka mirror-maker.
  • a
    that makes sense. so, just replicate into 1 or two locations, and have some way to switch read load?
  • m
    Once you have replicated, you can switch load, or have sticky routing (based on partition), etc
  • m
    Hello Pinot community, we at LinkedIn, are planning to host a Virtual Meetup for Apache Pinot on Aug 27. We would like to have one presenter from the community share their success-story/journey/etc with Pinot. Please reply to this thread if you would be interested in speaking, with a brief on what you would like to talk about.
    👍 3
  • m
    Is anyone using the 
    distinctCountThetaSketch
    aggregation in Pinot? The current syntax does not support 
    set-difference
    , and we might need to modify the last argument a bit. Evaluating if there's a backward compatibility we need to maintain.
  • k
    +1 for adding set-difference semantics
  • y
    I am querying a table in Pinot Query Console with id = 21400000008726127. the response it returns is id = 21400000008726130. The same if I do through broker, it is working fine.
  • t
    Hello, does Pinot support pagination when Group By is specified? These two queries give me the same results For the second one the pagination is ignored.
    Copy code
    select foo_name, AVG(foo_age) from fooTable group by foo_name order by foo_name DESC limit 5
select foo_name, AVG(foo_age) from fooTable group by foo_name order by foo_name DESC limit 5,5
    Pagination works for simple selection without a group by. These 2 queries give me different results and the merged results are the same as doing a similar query with no pagination and a limit of 10.
    Copy code
    select foo_name, foo_age from fooTable order by foo_name DESC limit 5
select foo_name, foo_age from fooTable order by foo_name DESC limit 5,5
  • t
    Ok. Thanks for confirming
1...140141142...160Latest