Hi, we’ve been trying to setup Apache Pinot ingestion from Azure EventHub (Azure EventHub supports Kafka protocol so that Kafka clients can read from Azure EventHub). All the events within Azure EventHub are Protobuf events. Any ideas on how to make protobuf deserialization work here? The table creation is successful but data is not being ingested.

Hi, does anyone know why I might have a weird / corrupted ideal state after moving a realtime tables to a different set of servers (via rebalance)? The table looks fine from the UI (all the segments are Good, it says they are all on the new servers, it can be queried) but I think there are issues when I try to do things like reload all the segments.

Is anyone else using Pinot’s Hadoop map-reduce support for building segments? Asking because after switching to 0.8 (from 0.7.1) it no longer works (issue with not finding HDFS plugin), plus some other odd issues.

Does pinot have a default limit?

select id from table group by id limit 100000000

only returns 1.1 M rows from API and JDBC client.

Hey, Wanted to ask which Pinot distribution is recommended for production deployments in Kubernetes cluster? Are there any other recommendations for production deployments?

Hi team, for Boolean type of field, what’s the main difference to define it as a Boolean vs string in the schema, in terms of querying the table?

Excerpt from log file : 2021/08/30 061644.488 ERROR [StatusUpdateUtil] [HelixTaskExecutor-message_handle_thread] Exception while logging status update org.apache.helix.HelixException: HelixManager (ZkClient) is not connected. Call HelixManager#connect()         at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:363) ~[pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at org.apache.helix.util.StatusUpdateUtil.logMessageStatusUpdateRecord(StatusUpdateUtil.java:348) [pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at org.apache.helix.util.StatusUpdateUtil.logError(StatusUpdateUtil.java:400) [pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:359) [pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.9.0-SNAPSHOT-jar-with-dependencies.jar:0.9.0-SNAPSHOT-2302bd2c01655d803e96e825143f03c675ed32ff]         at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?

Capture3.PNG

This message was deleted.

Hi, I am trying to run the latest version of Pinot with Docker (Mac) and the container suddenly stops. I don’t see any errors in the log nor are there any other containers running at that time.

docker run \
    --network=pinot-demo \
    --name pinot-quickstart \
    -p 9000:9000 \
     apachepinot/pinot:latest QuickStart \
    -type batch

Does anyone have any idea what might be going on? Please find attached logs

I'm tasked with doing a Pinot POC for my organization, as we're considering switching to it as our primary data store for reporting data. I followed the Advanced Pinot Setup guide and was able to create a realtime table ingesting streaming github events. I'm now trying to setup my own realtime table ingesting dummy data with a JSON column and UPSERTs enabled (this will be required for our use case). I have successfully uploaded both a table config and a schema to the pinot controller, and I also created a little app to push dummy data into a Kafka topic. I confirmed that the data is successfully being added to the topic, however my table is not ingesting any records. Can someone help me troubleshoot why that may be happening? I will post the table config and schema in this message's thread

I have a realtime table ingesting from Kafka and an application that writes events to the appropriate Kafka topic. My table originally had

realtime.segment.flush.threshold.rows=30

, and I ran my application to the point where I had pushed around 100k rows to a kafka topic before realizing that this was much too small a segment size. I stopped my app, deleted the table, changed

realtime.segment.flush.threshold.rows=100000

, and recreated it. Then, I ran my app to push 3mil rows to the kafka topic. At some point in Pinot's ingestion process, the status of my table changed to

BAD

. I looked in the controller logs and noticed this error:

2021/09/02 16:02:48.585 ERROR [SegmentCompletionFSM_simplejson__0__632__20210902T1602Z] [grizzly-http-server-21] Caught exception while committing segment metadata for segment: simplejson__0__632__20210902T1602Z

java.lang.IllegalStateException: Failed to find IdealState for table: simplejson_REALTIME

In the web UI for the Pinot Controller, under the Cluster Manager for the affected table, I sorted the

SEGMENTS

list by Status and noticed that I had two

Bad

segments. Inspecting the bad segments, I noticed that each had a total of 30 documents. I checked a handful of

Good

segments, and each had 100k documents. I'm not sure how to bring these segments into a

Good

state, or why they entered into a

Bad

state in the first place. I was unable to find anything in Pinot's documentation on what causes this error or how to resolve it.

Hi team, we are seeing lots of “BrokerResourceMissingError”, based the source code, this can happen only when table queried is not found? Is there any other case can result in this exception e.g. server or broker is under heavy load etc?

Hi folks, is there any way of using environment variables in the Pinot Ingestion config. I’ve checked this PR https://github.com/apache/pinot/pull/6271, but I don’t know how to setup the env variables. Thanks in advance.

how to check timezone of pinot from dashboard ?

I have a new dev machine and am trying to build Pinot. While the build succeeds, the compiler isn’t able to find this class:

org.apache.pinot.common.proto.Server

. This is generated from

server.proto

. Do I need to do anything other than

mvn clean install -DskipTests -Pbin-dist

as documented?

Hi team, there is a performance issue when using ‘JSONPATH’ transformation functions, this is may be caused by the jayway Inefficient LRUCache design. When encountering high concurrency in data ingestion, there will be fierce competition for CPU and all ingestion threads are waiting for the lock, and data consumption will be delay.

I can see the pinot-zookeeper disk usage high .. what could be the root cause since metadata cant be 65 gb out of 95 gb given with few million records in table .. is it database indexing causing some external views to be stored on zookeeper disk? @Mayank @Xiang Fu @Jackie @Subbu Subramaniam @Kishore G

I'm trying to use tenant tags with the kafka low level consumer to split consuming v consumed partitions across servers. However the offline servers don't seem to be getting any segments. Are there additional steps needed to get this to work. Am also using upsert functionality.

We are running pinot in kubernetes, and noticed that the servers are considered ready too early, before the server has managed to start. This causes the statefulset rolling restart to restart multiple servers simultaneously, making segments inaccessible. The server api

/health

endpoint should be used for readiness probing?

Hi everyone, I have an issue with a k8s deployment. Basically controllers are discovered twice: once via headless service and one more time via regular service. The one discovered through the regular services is always reported as "failed" as there is no ZK entry with the FQDN of the service. Is there any way to fix this?

Hi team, recently we had experienced Pinot server out of memory issue in a 4 server Pinot cluster when issuing one query select distinct id on entire table with only 1 billion record. We had to manually restart the Pinot servers pods to recover. Is this normal? Is there some index or Pinot cofigurariom we can add to this id or Pinot cluster to prevent it bring down the entire cluster.

hello, i could use some help with query tuning:

# schema:
user(int) | location(int) | time(long)

# 1st query (filter):
"select user, count(*) from {table} where time between {start} and {end} and location between 500 and 550 group by user having count(user) >= 24 limit 1000000"

# 2nd query (combiner):
"select time, count(distinct(user)) as count from {table} where user in ({users}) and time between {start} and {end} and location between 300 and 350 group by time limit 10000000"

the query time scales linearly with the number of selected userids from the first query

while setting up this config in local machine , it is working fine .

Hi can you remove the enable slider button from the web UI

Are there any guides for sizing pinot, e.g. number of servers, brokers, controllers. Seems my setup has stopped working well with most queries only getting a response from one server instead of expected 2. I have added memory to servers, clustered zookeeper, ran recommendation engine for tables but seems not much is making a good improvement.

Hi team, I found that Pinot sometimes shows that the table status is bad, and at the same time the corresponding segment is also bad (one replicas state is consuming, and the other is online), this bad state will be automatically recovered. So when querying, especially on-demand will the query get inconsistent results, or how will Broker choose which replicas to query when querying?

Hello team, does Pinot need any system dependencies to run unit tests?

hey, does anyone have any recommendations in a production env as to what to do around capacity for the pinot-server, I’m working on a proof of concept with some real time data, and i already filled up my disk space, what are some of the things we can do to mitigate increasing disk space. I have been reading about this stuff: https://docs.pinot.apache.org/operators/operating-pinot/tiered-storage https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows for now i’m gonna increase my disk size but chances are that i’m gonna fill it up eventually lol, what are some of the things we can do to save disk space?

The segments are not being copied to s3.I get this message

Moved segment airlineStats_batch_2014-01-01_2014-01-01 from temp location /tmp/pinot-tmp-data/fileUploadTemp/tmp-66a57920-be10-41a0-a5e3-3f752b660d7c to /var/pinot/controller/data,<s3://ca-ctct-transient-dev-us-east-1-eigi-datalake/pinot-data/pinot-s3-example/controller-data/airlineStats/airlineStats_batch_2014-01-01_2014-01-01>