Hi! I was wondering, is there any easy-enough way to estimate Java heap usage for streaming upsert? For example, if I have 300 million primaries keys, is there a way to estimate how much heap space I’ll need?

Hello ,Has anyone successfully established connection from Tableau to Pinot? Please help

Hiii all! Do we have any Tableau experts in the audience? 🙂

Need some advise on how to use Date column in Tableau live when connected to Pinot

and need some advise on how to publish data in tableau server

Hey Team, Need some advice to solve following use case. Need to pass integer values as filter in IN clause. Number of values can range upto 5-20k. How can we apply such filter in Pinot Query? This filter values cannot be persisted in pinot table since this Dimension table can change and cannot be queried on Old state. So values are passed on by request itself. Came across, filter ID_SET. but required data in table. Thanks

Is there active development going on Tableau support so that we get JDBC drivers fully functional ?If so when

Hi, we are fighting a problem with bad segments and struggling to work out what is upsetting Pinot. The error we are getting is:

"Exception while executing a state transition task transaction_record_new_OFFLINE_1568570190761115000_1568576017322601000_3
java.lang.reflect.InvocationTargetException
       at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.base/java.lang.reflect.Method.invoke(Method.java:566)
       at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)
       at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)
       at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
       at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException
       at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:429)
       at org.apache.pinot.segment.local.segment.index.readers.forward.BaseChunkSVForwardIndexReader.<init>(BaseChunkSVForwardIndexReader.java:72)
       at org.apache.pinot.segment.local.segment.index.readers.forward.FixedByteChunkMVForwardIndexReader.<init>(FixedByteChunkMVForwardIndexReader.java:40)
       at org.apache.pinot.segment.local.segment.index.readers.DefaultIndexReaderProvider.newForwardIndexReader(DefaultIndexReaderProvider.java:104)
       at org.apache.pinot.segment.spi.index.IndexingOverrides$Default.newForwardIndexReader(IndexingOverrides.java:205)
       at org.apache.pinot.segment.local.segment.index.column.PhysicalColumnIndexContainer.<init>(PhysicalColumnIndexContainer.java:166)
       at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:181)
       at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:121)
       at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:91)
       at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.addSegment(OfflineTableDataManager.java:52)
       at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:373)
       at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:355)
       at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:162)
       ... 11 more

The segment is created with minion

SegmentGenerationAndPushTask

from a avro file in google bucket.

Hi Team, Need some clarity over how pagination is currently working with SELECTION (https://docs.pinot.apache.org/users/user-guide-query/querying-pinot#pagination-on-selection). Pagination happens on Broker , after bringing whole data from servers?

question… what happens if a really old event gets thru a realtime table? wonder what the implications of it are in a hybrid setup, would that record be lost in the ether?

Hi team, is there a reason that SEGMENT_UPLOAD_TIMEOUT is not configurable?

I am seeing a strange query perf issue: the following query should be blazingly fast - instead it is taking 10seconds +: select distinct($segmentName) from <table_name> where ts > 1659083681436 AND ts < 1659093681436 limit 200 Given that broker already has segment pruning logic based on ts, I am not sure and ts column has a range index (no dictionary), I am confused why this query is so slow.

When I do explain plan for this, I see that it is using scan based filter - why is this query not using range index?

Hi Team, I met a problem. For a custom r2o task, after filtering some data, the number of doc in the segment has decreased as expected, but it is found that the generated segment has also become larger. When viewing the segment, it is found that the columns.psf file has become larger. Excuse me, what is the possible reason? ?

This message was deleted.

When a segment file fails to be uploaded to the deep store (here: https://github.com/apache/pinot/blob/30c4635bfeee88f88aa9c9f63b93bcd4a650607f/pino[…]he/pinot/core/data/manager/realtime/PinotFSSegmentUploader.java), the tar.gz file gets removed. Does this segment ever get retried? I do have

controller.realtime.segment.deepStoreUploadRetryEnabled=true

but have not seen it happen. This is a controller setting, but it’s the server doing the upload.

Our RealtimeToOfflineSegmentsTask seems to be stuck and I'm having a hard time diagnosing why. The controller repeatedly reports: Found incomplete tasks: [Task_RealtimeToOfflineSegmentsTask_1659285958877] for same table: immutable_events_REALTIME. Skipping task generation.

Hello team. Running into this exception as I create Pinot Real time table :

Caught exception in state transition from OFFLINE -> ONLINE for resource: <redacted>, partition: <redacted>__21__0__20220801T1624Z
java.lang.IllegalStateException: Unable to create temp resources directory at /var/pinot/server/data/index/tmp. Check that the user has permissions on this directory.
	at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.core.data.manager.BaseTableDataManager.init(BaseTableDataManager.java:117) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.core.data.manager.offline.TableDataManagerProvider.getTableDataManager(TableDataManagerProvider.java:70) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.createTableDataManager(HelixInstanceDataManager.java:180) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.lambda$addRealtimeSegment$1(HelixInstanceDataManager.java:169) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705) ~[?:?]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:169) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:86) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at jdk.internal.reflect.GeneratedMethodAccessor316.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Hi team, I’m trying to use deep store with split commits (server uploads). Currently they are failing due to timeout errors (still need to investigate this on our end) but I’m seeing the following log messages in the controller log:

2022/07/29 17:37:10.353 INFO [LLCSegmentCompletionHandlers] [grizzly-http-server-38] Response to segmentCommitEndWithMetadata for segment:events__2__0__20220729T1717Z is:{"offset":-1,"status":"COMMIT_SUCCESS","isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1}

I’m just wondering why it says

isSplitCommitType":false

when both controller and server configs specify

pinot.server.instance.enable.split.commit=true

and

controller.enable.split.commit=true

I tried to build the source code with mvn option -

Djdk.version=8.

(https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally) i am facing errror Caused by: java.lang.ClassNotFoundException: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner.. the jar built doesnt have this class. Any suggestions here?

Hi, I've noticed that using regex in TEXT_MATCH ends up getting different results from using REGEXP_LIKE. It appears that TEXT_MATCH sometimes misses data. Is this behavior expected?

hello all, how to do incremental loading in Pinot

A rather noob question about ingesting into Pinot from kafka. If the upstream (kafka) has bad data that’s causing ingestion to fail, does Pinot ignore the record and continue ingesting the remaining records, or does ingestion get into a bad state and stop altogether?

Hi! I was wondering, is it possible to do streaming upserts on a hybrid table? Or is it realtime only

is there an easy way to disable the pinot UI? I’m taking a look at this currently https://docs.pinot.apache.org/operators/tutorials/authentication-authorization-and-acls but wondering if there’s anything on just disabling UI for everyone without impacting anything else. wondering what is the setup that people usually do with their cluster so that people cannot mess with your tables and stuff.

Hi Team 👋 , We have a Pinot cluster with jmx exporter configured. However, on the controllers, we can only see

ValidationMetrics

getting exposed along with some jvm metrics and not the

ControllerMetrics

. We have been using the jmx config file from the apache/pinot repository - pinot.yml and want to keep a track of metrics like

SEGMENTS_IN_ERROR_STATE

but they do not get exposed by the jmx exporter at the defined port. Can I please get some help in figuring out how all the metrics can be exported.

- pattern: "\"org.apache.pinot.common.metrics\"<type=\"ControllerMetrics\", name=\"pinot.controller.segmentsInErrorState.(\\w+)_(\\w+)\"><>(\\w+)"
  name: "pinot_controller_segmentsInErrorState_$3"
  labels:
    table: "$1"
    tableType: "$2"

i am facing issue with the spark submit command for the cluster mode.. For the client mode i was able to run fine.. I am following the https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark documentation… getting below error Exception in thread “main” java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0: “s3: command: export PINOT_VERSION=0.11.0 export PINOT_DISTRIBUTION_DIR=s3://roku-dea-dev/sand-box/suraj/apache-pinot-0.11.0-SNAPSHOT-bin spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master yarn --deploy-mode cluster --conf “spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins” --conf “spark.driver.extraClassPath=/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --conf “spark.executor.extraClassPath=/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --jars “${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar,${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar,${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar,${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --files s3://roku-dea-dev/sand-box/suraj/spark_job_spec.yaml local:///home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar -jobSpecFile spark_job_spec.yaml

Hi team, is there an endpoint on the broker that I can query using GET . I’m trying out some external health check Actually, is there an overview of all endpoints? I know of /query and /query/sql but they require a POST

Hello, our Pinot logs have stopped working, we have filebeat set up to read logs out of the

/var/log/system.log

file which is no longer being created/populated, does anyone have any idea why?

$ ll /var/log/system.log
ls: cannot access /var/log/system.log: No such file or directory

I’m pushing avro serialized data in binary format in kafka and want to create a realtime table on top of it. However, I’m getting the following error when using `org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder`:

Caused by: java.lang.IllegalStateException: Avro schema must be provided

When using

org.apache.pinot.plugin.inputformat.avro.KafkaAvroMessageDecoder

I get:

Caused by: java.lang.NullPointerException
at org.apache.pinot.plugin.inputformat.avro.KafkaAvroMessageDecoder.parseSchemaRegistryUrls(KafkaAvroMessageDecoder.java:303)

How should I provide avro schema urls? I can’t create an avro schema from UI. What else?