pls help

I am trying to connect

superset

to Pinot using

<pinot://pinot-broker:9000/query/sql?controller=http://pinot-controller:9000/>

But, I am getting the below error…

ERROR: (builtins.NoneType) None
(Background on this error at: <http://sqlalche.me/e/13/dbapi>)

Could anyone assist?

Hello ! 👋 I've got the following scenario : • Data is integrated in multiple batches per day (in an OFFLINE table) ◦ Batch 1: _01/01/2021 (data date) - DATA 1, DATA 3, DATA 6 -> `Segment_1(date=01/01/2021, data=[DATA 1, DATA 3, DATA 6])`_ ◦ Batch 2: _01/01/2021 (data date) - DATA 2, DATA 4, DATA 5 -> `Segment_2(date=01/01/2021, data=[DATA 2, DATA 4, DATA 5])`_ • Data must be available asap, so 2 separate segments are generated & ingested into Pinot • Some data needs to be corrected after the initial data ingestion, say DATA 1 & DATA 2 I know it is possible to replace segments but, how can we handle replacing data across multiple segments ? Can we generate a new segment with only the modified data and ignore old data in previous segments ? (

Segment_1

&

Segment_2

) ->

Segment_3(date=01/01/2021, data=[DATA 1, DATA 2])

Or do we have to regenerate the 2 segments entirely ? (if so, we need to identify what they contain) - Possibly after merging them ?

@Jack Do we have a doc to describe the preprocessing for partition/sort before ingestion? If so could you share? If not, could we add the doc? cc: @Syed Akram

I’ve been fooling around with how Pinot handles the “URI push” of segments. It seems like if I’m not using HDFS for deep storage, then the controller will download the segments before pushing to the server, which seems like it’s not a win. Is that correct? And (so far) I haven’t been able to configure the controller to successfully handle an HDFS URI push request, at least when I’m not using HDFS for deep storage - I see the msg when the controller starts up that the “hdfs file system” was initialized, but when it gets the URI push request, it fails with an error about the hdfs file system not being initialized. Any ideas?

Hi Everyone, Is there any column limits on Pinot table UI. I am loading Kafka topic data in Pinot table. I have only one row in Kafka topic. When I am trying to load 20 column for the same data is working file and I am able to see the correct data in pinot but when am trying with all the column (75 column) in pinot table it's showing 0 count. Kindly suggest.

Hi All I am trying to setup thirdeye using the standalone method and running this command

docker run \                                                  
    --network=pinot-demo \
    --name thirdeye \
    -p 1426:1426 \
    -p 1427:1427 \
    -d apachepinot/thirdeye:latest ephemeral

Giving this error

docker: invalid reference format.
See 'docker run --help'.
zsh: command not found: --network=pinot-demo

Please suggest how to proceed here

@Neha Pawar @Alexander Pucher @Ting Chen kindly suggest for one of the instance it's taking 8008 port is there any way to fixed it. I have checked configuration files.but couldn't find to update this pinot.server.netty.port value as 8098. I am not able to understand from where for one of the instance this is taking value as 8008 instead of 8098. I have restarted everything but still it's taking this value.

<!here> how can we increase pinot ingestion throughput if we are consuming messages from kafka topic . .say the output topic have only 1 partition .. will increasing partition of the kafka topic from where pinot is consuming automatically increase pinot throughput ... means pinot is smart enough to increase the no of consumers consuming from that topic and storing into tables/segments ..

I’m got this errors when query in pinot-server. Any idea how to resolve it ? I already delete all table & schema then recreate but it’s still errors.

Hi everyone, I am trying to access pinot tables in presto. I have created one pinot.properties file inside Presto catalog directory. pinot.properties connector.name= pinot pinot.controller-urls=Controller_10.187.157.33:9000 Then I m trying to start prestro with pinot. So used. ./presto --server Prestro_Host_Name:port --catalog pinot Now prestro is connected.here when I am doing show schemas; as I want to see the tables and schema. It's showing catalog 'pinot' does not exist. Kindly suggest. P.s. when I am doing show Catalogs then it's not showing there also. Kindly suggest if I need to do any changes or missed any step.

Pinot Newbie here. Batch example worked well. Trying quickstart Pinot Stream example. Could not access Pinot Data explorer - http://localhost:9000. Here is the command -

docker run     --network=pinot-demo     --name pinot-quickstart     -p 9000:9000     -d apachepinot/pinot:test QuickStart     -type stream

Here is the exception in the log - SSL handshake failed

Hi I found another bug for pinot : https://github.com/apache/incubator-pinot/issues/6948 Please help review thx

Hello 👋 What's the meaning of

{'errorCode': 410, 'message': 'BrokerResourceMissingError'}

? Got it from

pinot-db

Python client The query runs fine from the UI though

Hello, are transform configs supported when generating OFFLINE segments? I'm trying to add a new column via a date transformation and getting:

Caught exception while gathering stats
org.apache.parquet.io.InvalidRecordException: NEW_FIELD_NAME not found in message schema {

ingestionConfig:

"ingestionConfig": {
    "transformConfigs": [
      {
        "columnName": "NEW_FIELD_NAME",
        "transformFunction": "fromEpochDays(OLD_FIELD_NAME)"
      }
    ]
  },

is there a way to represent an array literal in pinot sql? I tried

ARRAY[1,2,3]

in a filter and a select and it didn't work.

Hi there, can someone help for monitoring apache Pinot?is there any api that Pinot exposes for metrics/dashboard by default. CC @Mohamed Sultan

Hello, I’m having issue with

LIMIT

on a table with 4.5 millions of rows, When I’m doing this query:

SELECT * FROM datasource_609bc4f74e3c000300131110 ORDER BY "timestamp" ASC LIMIT 100000,10

I’m getting a result in ~2.5s, and I can see in the query response stats

totalDocs=4794306

which is fine But when I’m doing this one (offset 1 000 000 instead of 100 000):

SELECT * FROM datasource_609bc4f74e3c000300131110 ORDER BY "timestamp" ASC LIMIT 1000000,10

I’m getting no rows (which isn’t the expected behavior) in ~10s and the totalDocs is

569840

(which isn’t the expected behavior either?) I have an hybrid table with segmentPruning by time Do you have any idea why I’m having this kind of issue? Thank you

Exception caught: java.lang.NullPointerException: null at java.io.Reader.<init>(Reader.java:78) ~[?:1.8.0_275] at java.io.InputStreamReader.<init>(InputStreamReader.java:97) ~[?:1.8.0_275] at org.apache.pinot.tools.admin.command.AbstractBaseAdminCommand.readInputStream(AbstractBaseAdminCommand.java:106) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at org.apache.pinot.tools.admin.command.AbstractBaseAdminCommand.sendRequest(AbstractBaseAdminCommand.java:101) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at org.apache.pinot.tools.admin.command.PostQueryCommand.run(PostQueryCommand.java:144) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at org.apache.pinot.tools.admin.command.PostQueryCommand.execute(PostQueryCommand.java:150) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] getting above exception while running a query, what could be the possibility?

Hello, How is someone meant to configure a json index by a string field that may sometimes be missing? I'm getting the following exception:

java.lang.IllegalStateException: Cannot flatten value node: null
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.spi.utils.JsonUtils.flatten(JsonUtils.java:246) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.realtime.impl.json.MutableJsonIndex.add(MutableJsonIndex.java:71) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:631) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:475) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]

Because a given field comes with null default value:

"inputForUiControls" : "null",

It's schema definition is:

dimensionFieldSpecs: [,...., {
      "name": "inputForUiControls",
      "dataType": "STRING",
      "maxLength": 2147483647
}]

Hello, can anyone share some light one on the root causes of the following exception:

2021/05/21 14:32:34.378 ERROR [SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread] Caught exception in state transition from OFFLINE -> ONLINE for resource: HitExecutionView_REALTIME, partition: HitExecutionView__12__21__20210520T1019Z
org.apache.pinot.core.segment.index.loader.V3UpdateIndexException: Default value indices for column: inputForUiControls cannot be updated for V3 format segment.
	at org.apache.pinot.core.segment.index.loader.defaultcolumn.V3DefaultColumnHandler.updateDefaultColumn(V3DefaultColumnHandler.java:53) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.core.segment.index.loader.defaultcolumn.BaseDefaultColumnHandler.updateDefaultColumns(BaseDefaultColumnHandler.java:144) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.core.segment.index.loader.SegmentPreProcessor.process(SegmentPreProcessor.java:104) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:99) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:283) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:138) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]

I've changed the default value of a String dimension field from

"null"

to

"{}"

I got the following exception when trying to consume from kafka using high level consumer, I checked the configured

stream.kafka.hlc.bootstrap.server

,

stream.kafka.hlc.zk.connect.string

,

stream.kafka.zk.broker.url

is correct, but it’s not consuming.

2021/05/21 13:59:59.025 WARN [PinotRealtimeSegmentManager] [<http://ZkClient-EventThread-104-mse-93c7add0-zk.mse.aliyuncs.com:2181|ZkClient-EventThread-104-mse-93c7add0-zk.mse.aliyuncs.com:2181>] Caught exception while processing segment fetrace_biz2-pinot_0__0__1621605599020 for instance Server_pinot-server-2.pinot-server-headless.pinot.svc.cluster.local_8098, skipping.
java.lang.NullPointerException: null
        at org.apache.pinot.controller.helix.core.realtime.PinotRealtimeSegmentManager.assignRealtimeSegmentsToServerInstancesIfNecessary(PinotRealtimeSegmentManager.java:247) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-255202ec4fc7df2283f7c275d8e9025a26cf3274]
        at org.apache.pinot.controller.helix.core.realtime.PinotRealtimeSegmentManager.processPropertyStoreChange(PinotRealtimeSegmentManager.java:304) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-255202ec4fc7df2283f7c275d8e9025a26cf3274]
        at org.apache.pinot.controller.helix.core.realtime.PinotRealtimeSegmentManager.handleDataChange(PinotRealtimeSegmentManager.java:405) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-255202ec4fc7df2283f7c275d8e9025a26cf3274]
        at org.apache.helix.manager.zk.zookeeper.ZkClient$7.run(ZkClient.java:1039) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-255202ec4fc7df2283f7c275d8e9025a26cf3274]
        at org.apache.helix.manager.zk.zookeeper.ZkEventThread.run(ZkEventThread.java:69) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-255202ec4fc7df2283f7c275d8e9025a26cf3274]

Hello, I have a Pinot deployment done through K8s wherein I'm progressively adding fields to a realtime table. This deployment is very basic (1 server instance only, with 6GB for java Heap, the pod has a memory limit of 7GB, 100GB persistent storage, deep storage for segments has been enabled) but I'm getting multiple server restarts because the pod keeps getting killed with OutOfMemory errors while ingesting data and creating segments. It seems the cause is not the JVM itself but off-heap memory maps, please see the following image for more details.

Hi, Prometheus/grafana and pinot is deployed in two different namespace. Can we integrate that two and monitor?? We are not getting pinot log metrics in Prometheus. @Xiang Fu @Mayank

I think this may not be related to nullValue support. What does query console return?

The API returns

"-Infinity"

and the Web UI shows the same

controller.segment.fetcher.auth.token="Basic YWRtaW46dmVyeXNlY3JldA==" controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory controller.admin.access.control.principals=admin,user controller.admin.access.control.principals.admin.password=verysecret controller.admin.access.control.principals.user.password=secret controller.admin.access.control.principals.user.tables=baseballStats controller.admin.access.control.principals.user.permissions=READ controller.port=9000 controller.host=localhost controller.zk.str=localhost:2191 controller.data.dir=/home/sas/temp/rawdata controller.helix.cluster.name=QuickStartCluster

@Syed Akram how you enabled this authentication in Pinot.I am also trying to do that. Have you created new controller.properties file and while starting the controller used this file using --confFilename

@Alexander Pucher could we add docs on using/configuring ACLs in Pinot, or get a pointer if they already exist?