General question: Why is pinot broker/controller a statefulset in the helm chart? Those can be deployments isnt it? They dont have any persistence afaik.

If I want to move pinot table from one tenant to another, can I just change the tenant name inside table config? If yes, is there any downtime?

Hi team, I believe it is possible to apply inverted index on multiple columns. Any idea on how it is stored? All the documentation I have seen so far give examples of inverted index on only 1 column. I would like to apply inverted index on multiple columns in an optimized way.

Hi team, if two tables ingest the same Kafka topic, will they have the same copy of data?

What’s the best practice to set stream.kafka.consumer.prop.auto.offset.reset? I haven’t found detailed info about it. I set it smallest in case stream data lost. But I met a situation, when I restarted the controller, pinot just stopped consuming data.

Hi everyone! I’ve been doing POC on Pinot, and currently facing issue while ingestion orc file data to pinot. Filed an GH issue as well: https://github.com/apache/pinot/issues/8460 Can anyone help?

Does Pinot support using

pinot-jdbc-client

in JMeter and Perf testing queries performance of Pinot using jmeter’s JDBC Request Sampler?

Hey there, which metric can I use to check the consumption lag from the servers in comparison with the topic they are consuming from?

hello everyone i try to config tls-ssl on my apache pinot cluster. Still not working. Anyone can share how to config enable tls-ssl on pinot cluster controller, broker, server, minion... how to connect with tls-ssl to apache pinot.

Hi. Starting to get dirty hands on code to implement my GDPR purge process. I’m a bit struggling with the build time and the no way to test. Is there any way to get a faster build to test it ? I’ve used the folowing maven command

mvn install package -DskipTests -Pbin-dist

and it take more than 20 minutes to build 😕 Anyway to get faster ?

Hi there. Is there a way to transform a string field to a double/float upon ingestion? I'm looking at the supported transform functions and do not see a function that supports this https://docs.pinot.apache.org/users/user-guide-query/supported-transformations

Anybody know the differences in use cases between Pinot, druid, Clickhouse, and Rockset?

image.png

Hi Pinot folks, we observed a peculiar incident today wherein consumption stopped from just 1 partition of a topic (this topic has 96 partitions, 95 are working fine). This segment was moved from CONSUMING to OFFLINE state due to some exception during consumption.

0 2022/04/06 12:06:04.179 ERROR [LLRealtimeSegmentDataManager_span_event_view_1__50__287__20220406T1205Z] [span_event_view_1__50__287__20220406T1205Z] Exception while in work
  1 2022/04/06 12:06:04.365 INFO [FileUploadDownloadClient] [span_event_view_1__50__287__20220406T1205Z] Sending request: <http://controller-0.controller-headless.pinot.svc.cluster.local:9000/segmentStoppe>    dConsuming?reason=java.lang.NullPointerException&streamPartitionMsgOffset=1059610656&instance=Server_server-span-event-view-realtime-7.span-event-view-realtime-headless.pinot.svc.cluster.local_8098&of    fset=-1&name=span_event_view_1__50__287__20220406T1205Z to controller: controller-0.controller-headless.pinot.svc.cluster.local, version: Unknown
  2 2022/04/06 12:06:04.366 INFO [ServerSegmentCompletionProtocolHandler] [span_event_view_1__50__287__20220406T1205Z] Controller response {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"build    TimeSec":-1,"status":"PROCESSED","offset":-1} for <http://controller-0.controller-headless.pinot.svc.cluster.local:9000/segmentStoppedConsuming?reason=java.lang.NullPointerException&streamPartitionMsgO>    ffset=1059610656&instance=Server_server-span-event-view-realtime-7.span-event-view-realtime-headless.pinot.svc.cluster.local_8098&offset=-1&name=span_event_view_1__50__287__20220406T1205Z
  3 2022/04/06 12:06:04.366 INFO [LLRealtimeSegmentDataManager_span_event_view_1__50__287__20220406T1205Z] [span_event_view_1__50__287__20220406T1205Z] Got response {"isSplitCommitType":false,"streamParti    tionMsgOffset":null,"buildTimeSec":-1,"status":"PROCESSED","offset":-1}

I have attached the server logs when this happened.

Hello everyone, Im struggling to create a Kafka stream ingestion with HLC consumer and a custom consumer group id. Anyone has worked on a similar case ?

Hello everybody. Does Pinot support data export out of Pinot via bulk extract or CDC?

Hi All, do you know how Pinot stores data between brackets? (not JSON). Let me explain: I have this data

id,timestamp,application
1,1649268351,"{'app_name': 'foo', 'version': '1.0.0', 'app_id': None, 'business': 'ponzico'}"

And when I load that info I got

| id | timestamp  |    application    |
|----|------------|-------------------|
| 1  | 1649268351 | foo,1.0.0,ponzico |

(In table-schema.json row "_application_" is configured as "STRING".) There is a way to query "_application_" row based on one of the values inside? (ie

SELECT * FROM testing WHERE application.app_name = "foo"

). Thanks in advance!

Hey there! I have a different type of question this time: If I had to give a presentation to my company advocating for us to start using Pinot as a go-to tool for user-facing real-time analytics, which arguments or points of view you would recommend me to speak about?

Hi everyone! I was wondering whether the dataset used is available somewhere (Airline data from 1987-2008). I am currently working on my Master's thesis and I would like to run some experiments on that dataset. Thanks

Hi there, Checking community reviews. We are in process of setting up Kubernetes-based deployment of Pinot Cluster. Has anyone seen significant performance gains from using SSDs with instance store instead of EBS for server PODs?

Hi, is it supported to use JSONEXTRACTSCALAR function in the transformConfigs?

Hey guys, Is there a way to connect Pinot to a secure Kafka cluster using Jaas configuration?

Hi getting deeper in my purge process dev many thanks to the pinot team for the awesome support and code quality 😉 . Is there any way to upload a realtime segment ? Like purging it and refreshing it ? I do not really inderstand why this is not possible if the segment is not consuming 😕 And I’m a bit surprised of the API getting an OFFLINE table for the uploadProcess instead my original type REALTIME

Any plans for pinot to support 'RealtimeToOfflineSegementsTask' for upsert enabled tables. Use case is this: 1. Normal queries; key contains a business date; only want to see the last value for a day (as per current upsert config works well) 2. Detail / as of queries; want to see record up to a given timestamp for a given business date (skipUpsert and LASTWITHTIME( col, asOfTime ). e.g. compare 11am yesterday v today 3. Historical want to move data beyond a week to OFFLINE 4. Historical want to drop data points e.g. milli seconds -> hours per day. Like reducing data resolution over time through some data cleaning job.

It seems that we have health API for controller. Do we have health API for table and broker?

hey folks, does anyone have experience with “zombie” tables? I attempted to add a tableSpec but was missing a field in the config. This resulted in a null pointer exception when Pinot processed the config. As a result, the table doesn’t exist in the UI or by querying the list API, but I can’t create a table with the same name due to receiving an error that “table already exists”. Presumably the spec reached ZK or something, but no where else? What can be done to remove the zombie table?

Hi, does transformConfigs works when used along with complexTypeConfig in ingestionConfig section?

Hi, I checked Authorization in broker level.

pinot.broker.access.control.principals.<user>.tables=test_table

It seems that restarting broker is required whenever adding a privileged table. Do you restart every time you grant table privileges in the production environment ? Or is there any other way?

Hi Team, As part of a POC, We are trying to load pinot table data into a spark dataFrame using the spark JDBC option. However when we try we are seeing the following error:

Exception in thread "main" java.sql.SQLFeatureNotSupportedException
	at org.apache.pinot.client.base.AbstractBaseStatement.setQueryTimeout(AbstractBaseStatement.java:167)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:60)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:226)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:35)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:355)
	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
	at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)

When I add a table, the following error occurred. What’s the possible reasons for it? {“code”500,“error”“org.apache.kafka.common.KafkaException: Failed to construct kafka consumer”}