Apache Pinot #troubleshooting

Deena Dhayalan

09/15/2022, 10:36 AM

Hi all I get this error while doing a join query A simple Join Query

Copy code

select a.name from table1 as a JOIN table2 as b on a.name=b.name

Ali Atıl

09/16/2022, 12:13 PM

Hello everyone, Do dateTimeFieldSpecs columns have range index by default or do i have to define it in table configuration? in my use case, i am filtering on dateTimeFieldSpecs column and wonder if range index would make it faster?

Tiger Zhao

09/16/2022, 3:32 PM

Hi, we're seeing high disk usage on the minions from the ReatimeToOfflineSegments task. Is it expected that pinot will keep the segments on disk after finishing the task? Also we noticed that when running

lsof

, Pinot is keeping a large number of deleted segments open still, which might be some memory bug?

Luis Fernandez

09/16/2022, 4:58 PM

hello friends! we recently have been playing around with securing the pinot cluster, so far we just created some users and distributed the token among the different pinot components, things worked fine, however i have a question, once things get authorized, even though we authorize ourselves in the pinot controller UI if we try to use swagger we get forbidden errors, like swagger is expecting things to be authenticated i thought, since this is living in the controller and since we already put id and pw in the UI things would get authorized from there but i don’t think that’s the case how can we do authorized request from swagger? has anyone run into this similar issue? our workaround was to use postman and send the basic auth header ourselves but would like to do it thru swagger as it’s easier, thoughts?

🍷 1

erik bergsten

09/16/2022, 5:06 PM

When I turn on upserts in my table config I get the following error from the pinot-server:

Copy code

Caught exception while indexing the record: {
  "nullValueFields" : [ ],
  "fieldToValueMap" : {
    "deviceType" : "bar",
    "data" : "{\"test\":123}",
    "deviceId" : "foo",
    "ts" : 1663345155307
  }
}
java.lang.NullPointerException: null
        at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:515) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:573) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:430) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:623) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
        at java.lang.Thread.run(Thread.java:829) [?:?]

Using the following table config: https://pastebin.com/uXnyVSfx data schema: https://pastebin.com/yvLCa1cd Just changing the upsertConfig to null everything works. Any ideas?

Nagendra Gautham Gondi

09/16/2022, 6:30 PM

Hi, I am trying to create a Pinot Table that consumes the data from cross account Kinesis Streams. I have set environmental variables with credentials to get access to the cross account stream. However, I received an error in the corresponding pod that is trying to create the table: Error:

Copy code

{
  "code": 500,
  "error": "org.apache.pinot.shaded.software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException: Stream cross-account-stream under account xxxxxxxx8241 not found. (Service: Kinesis, Status Code: 400, Request ID: c3ed6e9c-d25b-dccd-9873-aee2d3c8f651, Extended Request ID: 0/0WUQm7Y1TiHJj0Xr207jlzuDDZw8iv9NyFrUHvHz5WH8kRxc6xNO0lZDxOCgzy8HyBAOzHrxW8F0097tbWUqOp0fTnCJRw)"
}

YAML File:

Copy code

apiVersion: v1
kind: ConfigMap
metadata:
  name: examples
  namespace: pinot-quickstart
data:
  CROSS_ACCESS_KEY_ID: "sample_key"
  CROSS_SECRET_ACCESS_KEY: "sample_secret_key"
  CROSS_SESSION_TOKEN: "sample_session_token"
  crossAccountEvents_realtime_table_config.json: |-
    {
      "tableName": "crossAccountEvents",
      "tableType": "REALTIME",
      "segmentsConfig": {
        "timeColumnName": "timeStampField",
        "retentionTimeUnit": "DAYS",
        "retentionTimeValue": "60",
        "schemaName": "crossAccountEvents",
        "replication": "1",
        "replicasPerPartition": "1"
      },
      "tenants": {},
      "tableIndexConfig": {
        "loadMode": "MMAP",
        "invertedIndexColumns": [
        ],
        "streamConfigs": {
          "streamType": "kinesis",
          "stream.kinesis.consumer.type": "lowlevel",
          "stream.kinesis.topic.name": "cross-account-stream",
          "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
          "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
          "realtime.segment.flush.threshold.time": "12h",
          "realtime.segment.flush.threshold.size": "100000",
          "stream.kinesis.consumer.prop.auto.offset.reset": "smallest",
          "region": "us-east-1",
          "shardIteratorType": "TRIM_HORIZON"
        }
      },
      "metadata": {
        "customConfigs": {}
      }
    }

  crossAccountEvents_schema.json: |-
    {
      "schemaName": "crossAccountEvents",
      "dimensionFieldSpecs": [
        {
          "name":"requestId",
          "dataType": "STRING",
          "defaultNullValue": ""
        },
        {
          "name": "accountId",
          "dataType": "STRING",
          "defaultNullValue": ""
        },
        {
          "name": "data",
          "dataType": "STRING",
          "defaultNullValue": ""
        }
      ],
      "dateTimeFieldSpecs": [
        {
          "name": "timeStampField",
          "dataType": "TIMESTAMP",
          "format": "1:MILLISECONDS:TIMESTAMP",
          "granularity": "1:MILLISECONDS"
        }
      ]
    }

---
apiVersion: batch/v1
kind: Job
metadata:
  name: cross-account-events-table-creation
  namespace: pinot-quickstart
spec:
  template:
    spec:
      containers:
        - name: cross-account-events-table-creation-json
          image: apachepinot/pinot:latest
          args: [ "AddTable", "-schemaFile", "/var/pinot/examples/crossAccountEvents_schema.json", "-tableConfigFile", "/var/pinot/examples/crossAccountEvents_realtime_table_config.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
          env:
            - name: JAVA_OPTS
              value: "-Xms4G -Xmx4G -Dpinot.admin.system.exit=true"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                configMapKeyRef:
                  key: CROSS_ACCESS_KEY_ID
                  name: examples
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                configMapKeyRef:
                  key: CROSS_SECRET_ACCESS_KEY
                  name: examples
            - name: AWS_SESSION_TOKEN
              valueFrom:
                configMapKeyRef:
                  key: CROSS_SESSION_TOKEN
                  name: examples
          volumeMounts:
            - name: examples
              mountPath: /var/pinot/examples
      restartPolicy: OnFailure
      volumes:
        - name: examples
          configMap:
            name: examples
  backoffLimit: 100

Can you please help me in resolving this error?

Huaqiang He

09/19/2022, 1:42 PM

Hi team, I get a query result that is very confusing. Can anyone help to reason it? query 1 uses a condition

"timestamp" between 1662946380000 and 1662946440000

and gets 985 items. query 2 uses a very close condition

add("timestamp", latency) between 1662946380000 and 1662946440000

and also gets 985 items. (

latency

is non negative) query 3 tries to combine the two conditions together

("timestamp" between 1662946380000 and 1662946440000 or add("timestamp", latency) between 1662946380000 and 1662946440000)

, but the result shows 18253149 items. I expect the result is between 985 and 985+985.

Copy code

1. select count("*") from api_requests where "timestamp" between 1662946380000 and 1662946440000 and entity_id='8a3688f4-6390-0e77-0163-d010adc10427'
count(*)
985

2. select count(*) from api_requests where add("timestamp", latency) between 1662946380000 and 1662946440000 and entity_id='8a3688f4-6390-0e77-0163-d010adc10427'
count(*)
985

3. select count("*") from api_requests where ("timestamp" between 1662946380000 and 1662946440000 or add("timestamp", latency) between 1662946380000 and 1662946440000) and entity_id='8a3688f4-6390-0e77-0163-d010adc10427'
count(*)
18253149

Deena Dhayalan

09/19/2022, 5:08 PM

Hi team , I am getting error while restarting server after added cluster config to enable v2 engine While starting broker itself , along with broker ,8442 port is starting

Copy code

Sep 19, 2022 10:28:47 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8097]
Sep 19, 2022 10:28:47 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
2022/09/19 22:28:49.770 ERROR [StartServiceManagerCommand] [Start a Pinot [SERVER]] Failed to start a Pinot [SERVER] at 6.81 since launch
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:8442
	at org.apache.pinot.server.worker.WorkerQueryServer.start(WorkerQueryServer.java:94) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.server.starter.ServerInstance.startQueryServer(ServerInstance.java:211) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.server.starter.helix.BaseServerStarter.start(BaseServerStarter.java:576) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:167) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:97) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
Caused by: java.lang.RuntimeException: java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:8442
	at org.apache.pinot.query.mailbox.channel.GrpcMailboxServer.start(GrpcMailboxServer.java:62) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.mailbox.channel.ChannelManager.init(ChannelManager.java:50) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.mailbox.GrpcMailboxService.start(GrpcMailboxService.java:65) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.runtime.QueryRunner.start(QueryRunner.java:90) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.service.QueryServer.start(QueryServer.java:64) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.server.worker.WorkerQueryServer.start(WorkerQueryServer.java:92) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	... 7 more
Caused by: java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:8442
	at io.grpc.netty.shaded.io.grpc.netty.NettyServer.start(NettyServer.java:328) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at io.grpc.internal.ServerImpl.start(ServerImpl.java:183) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at io.grpc.internal.ServerImpl.start(ServerImpl.java:92) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.mailbox.channel.GrpcMailboxServer.start(GrpcMailboxServer.java:60) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.mailbox.channel.ChannelManager.init(ChannelManager.java:50) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.mailbox.GrpcMailboxService.start(GrpcMailboxService.java:65) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.runtime.QueryRunner.start(QueryRunner.java:90) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.query.service.QueryServer.start(QueryServer.java:64) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	at org.apache.pinot.server.worker.WorkerQueryServer.start(WorkerQueryServer.java:92) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-2d6665b8e5fa0842ef67b3d9896c5e04ecad78e9]
	... 7 more
Caused by: io.grpc.netty.shaded.io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

Alice

09/20/2022, 10:46 AM

Hi team, got a error, any idea of root cause for it?

Copy code

Caused by: org.apache.lucene.queryparser.classic.ParseException: Encountered "<EOF>" at line 1, column 53.
Was expecting one of:
    <NOT> ...
    "+" ...
    "-" ...
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <NUMBER> ...
    
	at org.apache.lucene.queryparser.classic.QueryParser.generateParseException(QueryParser.java:931) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-3efcf3f00eaa6883eee62a111eee334fc330b72c]
	at org.apache.lucene.queryparser.classic.QueryParser.jj_consume_token(QueryParser.java:813) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-3efcf3f00eaa6883eee62a111eee334fc330b72c]
	at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:252) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-3efcf3f00eaa6883eee62a111eee334fc330b72c]
	at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:215) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-3efcf3f00eaa6883eee62a111eee334fc330b72c]
	at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:109) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-3efcf3f00eaa6883eee62a111eee334fc330b72c]
	at org.apache.pinot.segment.local.realtime.impl.invertedindex.RealtimeLuceneTextIndex.getDocIds(RealtimeLuceneTextIndex.java:106) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-3efcf3f00eaa6883eee62a111eee334fc330b72c]
	... 30 more

abhinav wagle

09/20/2022, 7:58 PM

Hi we are trying to run Pinot index Text_match and running into following issue with escaping a special char

. Our field value includes the

. Whats the right way to escape it. For example :

Copy code

TEXT_MATCH(active_exp_map, '/.*9DtZjvh\-R/') - returns nothing.
vs
TEXT_MATCH(active_exp_map, '/.*9DtZjvh/') - returns data

Lars-Kristian Svenøy

09/20/2022, 8:50 PM

Hey guys. Seeing an issue with dynamic table queries through the Pinot connector for Trino version 379. When doing a query with a WHERE clause on the partition key without using a dynamic table query where there is no data for that key, 0 rows are returned (as expected). But when doing a dynamic table query of the same kind, the Pinot Trino connector hangs for a long time, and then returns

Copy code

Query 20220920_202500_00441_6rt9p failed: Only 0 out of 0 servers responded for query select

This seems like a bug in the connector to me, any ideas?

Josh Clum

09/20/2022, 9:25 PM

Has anybody noticed an index out of bounds exception when trying to create a native text index over a column?

Nagendra Gautham Gondi

09/20/2022, 9:56 PM

Hi, I have created an EKS cluster in account A and would like to access kinesis data streams in different accounts. I am successful in passing the temporary credentials and accessing the cross account resource. However, is there any simpler way to manage accessing cross account streams without updating the env variables everytime?

Ali Atıl

09/21/2022, 6:47 AM

Hello everyone, Is ST_DISTANCE function not supported for V2 Engine? i am having an exception when i run a query with ST_DISTANCE and V2 Engine enabled. Query:

select lon,lat from mytable where ST_DISTANCE(location_st_point, ST_Point(-122, 37, 1)) < 5000 limit 1000

Exception:

[

{

"message": "SQLParsingError\njava.lang.RuntimeException Error composing query plan for: select lon,lat from mytable where ST_DISTANCE(location_st_point, ST_Point(-122, 37, 1)) < 5000 limit 1000\n\tat org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:131)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:147)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:125)\n\tat org.apache.pinot.broker.requesthandler.BrokerRequestHandler.handleRequest(BrokerRequestHandler.java:47)\n...\nCaused by: org.apache.calcite.runtime.CalciteContextException: From line 1, column 88 to line 1, column 108: No match found for function signature ST_Point(<NUMERIC>, <NUMERIC>, <NUMERIC>)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)\n...\nCaused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature ST_Point(<NUMERIC>, <NUMERIC>, <NUMERIC>)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)",

"errorCode": 150

}

]

Bobby Richard

09/21/2022, 5:39 PM

Is it possible to control the TTL of the keys kept in memory for Pinot deduplication? i.e. My table's retention is set to 90 days, but dupes are only possible for the initial 24 hours. I'd like to enable deduplication for the first 24 hours so I don't have to pay the price of keeping the keys of 90 days worth of data in memory.

🍷 1

Tanmesh Mishra

09/22/2022, 12:51 AM

I am trying to push my changes in the pinot-docs repo but I am getting this error.

Copy code

remote: Permission to pinot-contrib/pinot-docs.git denied to tanmesh.
fatal: unable to access '<https://github.com/pinot-contrib/pinot-docs.git/>': The requested URL returned error: 403

Do I need any read/ write permission ? Will really appreciate any suggestion resolving this issue.

sunny

09/22/2022, 6:30 AM

After adding a Kafka partition, there is a issue that data is not visible when querying the pinot partition table with a where query.

Copy code

1. Create Pinot Partitioned Table (kafka topic partition = 3)
- Query Successed.

2. Add kafka partition to a topic (3->4)
- New consuming segment(3__0) is created in pinot.

3. Add kafka topic data to a new partition.
- Query Successed (select * from)
- But, the row is not shown in the query ( select * from where in)
- The row is visible only when the segment is completed. However, the data coming into the new consuming segment(3__1) doesn't look the same as before.

It may happen that kafka partitions are increased during operation. So Please check the issue. 😊

abhinav wagle

09/22/2022, 5:29 PM

Hello if we use our zookeeper server, is this the only place we provide the zookeeper url : https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/values.yaml#L521 . Is there any other config/property we need to set for each pinot server, controller, broker so that Pinot pods can talk to zookeeper.

Luis Fernandez

09/22/2022, 6:49 PM

hello friends, we are going to start studying upgrading our clusters to pinot 0.11.0 I was wondering if there’s any gotchas that you all had regarding that upgrade and if following this https://docs.pinot.apache.org/operators/operating-pinot/upgrading-pinot-cluster still is the way to go and if anything change from going from 0.10.0 to 0.11.0 in terms of how to roll the upgrade, we did 0.9.0 to 0.10.0 following the doc and everything went alright but just wanna make sure there are no new unexpected things ahead of us

Tanmesh Mishra

09/22/2022, 11:42 PM

May I get some helps in resolving this issue. My test for this PR is failing even though it is successful in local testing. Any suggestion would be really helpful.

Eric Asuncion

09/23/2022, 1:52 AM

Running pinot locally on Mac M1 and noticed error below running “./bin/pinot-admin.sh QuickStart -type batch”: Session establishment complete on server localhost/000000012123, sessionid = 0x10017cb1fe20000, negotiated timeout = 30000 zkclient 0, zookeeper state changed ( SyncConnected ) ZkClient AsyncCallback retry thread is interrupted. Terminate ZkClient AsyncCallback retry thread. Interrupted waiting for success java.lang.InterruptedException: null at java.lang.Object.wait(Native Method) ~[?:?] at java.lang.Object.wait(Object.java:328) ~[?:?] at org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks$DefaultCallback.waitForSuccess(ZkAsyncCallbacks.java:248) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-7af0d20aaa9e8ec7ad652a0209457cd226a2b0f0] at org.apache.helix.zookeeper.zkclient.ZkClient.issueSync(ZkClient.java:1616) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-7af0d20aaa9e8ec7ad652a0209457cd226a2b0f0] at org.apache.helix.zookeeper.zkclient.ZkClient.access$300(ZkClient.java:85) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-7af0d20aaa9e8ec7ad652a0209457cd226a2b0f0] at org.apache.helix.zookeeper.zkclient.ZkClient$4.run(ZkClient.java:1642) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-7af0d20aaa9e8ec7ad652a0209457cd226a2b0f0] at org.apache.helix.zookeeper.zkclient.ZkEventThread.run(ZkEventThread.java:99) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-7af0d20aaa9e8ec7ad652a0209457cd226a2b0f0] zkclient 0, sycnOnNewSession with sessionID 10017cb1fe20000 async return code: OK and proceeds Closing zkclient uid:0, zkStateCONNECTED Timeout:30000 sessionid:0x10017cb1fe20000 local/00000001:52316 remoteserverlocalhost/00000001:2123 lastZxid:1 xid:2 sent:2 recv:2 queuedpkts:0 pendingresp:0 queuedevents:0 Session: 0x10017cb1fe20000 closed Closed zkclient with uid:0 I don’t recall seeing this error message before. Also, pinot terminates running “-type hybrid” but have not debugged it yet. Any plans to replace (or make it configurable) port 7000? I have to disable airplay every time.

Tommaso Peresson

09/23/2022, 10:00 AM

Hello, I have a general question for you. Did anybody ever used gcs as PinotFS through a proxy with spark ingestion? It seems like this library used in

pinot-gcs

doesn't support proxies and currently my spark workers can't connect to gcs. I've already tried to set

Copy code

-Dhttps.proxyHost -Dhttps.proxyPort

but they affect only the driver. Thanks

Stuart Millholland

09/23/2022, 1:40 PM

Hello all. We just upgraded one of our test k8s namespaces from .10 to .11 by deleting all of the statefulsets (with cascade=orphan) then deploying, then deleting the pods zookeeper, broker, controller, server, minion pods in that order and then re-deploying. Pinot was upgraded successfully but our segments were not downloaded from the deepstore. Looks like zookeper doesn't have knowledge of them. We know we can use the LaunchDataIngestionJob to load the deepstore segments (BTW this is a hybrid table), but I'm curious what we would do in production. Was there a step we missed that would have made pinot recognize the segments in the deepstore and automatically load them?

Stuart Millholland

09/23/2022, 7:13 PM

Does anyone have a sample job spec for loading up segments using a file pattern? I tried this and it didn't recognize the pattern

Stuart Millholland

09/23/2022, 7:13 PM

Copy code

some stuff
outputDirURI: '<gs://blah/data/immutable_events>'
includeFileNamePattern: "glob:**/*1661990400000*"

Gerrit van Doorn

09/23/2022, 8:56 PM

Hi folks, I’m trying to upgrade from release-0.10.0 to release-0.11.0 and I’m greeted with the following errors:

Copy code

2022/09/23 20:53:11.493 WARN [ClientCnxn] [main-SendThread(localhost:2185)] Session 0x3e3d835c1a2346f7 for server localhost/127.0.0.1:2185, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:356) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
2022/09/23 20:53:11.594 INFO [ZkClient] [main-EventThread] zkclient 5, zookeeper state changed ( Disconnected )
2022/09/23 20:53:12.441 INFO [ControllerResponseFilter] [grizzly-http-server-22] Handled request from 127.0.0.1 GET <http://localhost:9000/health>, content-type null status code 503 Service Unavailable

What could be the reason for this?

Neeraja Sridharan

09/26/2022, 7:35 PM

Hello team 👋 We have

offline tables in Pinot

with invertedIndexColumns, sortedColumn and segmentPartition (with Murmur based partitions) enabled. We also have instanceSelectorType as "replicaGroup". We've currently setup

createInvertedIndexDuringSegmentGeneration

flag to

false

by default. Is there a recommended approach to set this flag to

true

and also, what is the expected behavior? Will it be beneficial to enable it to minimize index creation after segments are loaded onto servers? Appreciate any help regarding this 🙇‍♀️

Piyush Mittal

09/27/2022, 9:52 AM

HI Team need to upgrade pinot to 0.11 version.Please suggest chart version to be used

Tommaso Peresson

09/27/2022, 1:02 PM

Hello everybody, I have a question for you. Is it possible to modify the metadata of a segment? I would like to: • create the segments with spark and store them in hdfs • move them with distcp to gcs • load them with a metadata push to the cluster but this leaves me with segments having

"custom.map": "{\"input.data.file.uri\":\"hdfs://***\"}",

and instead I would want to have something like

"custom.map": "{\"input.data.file.uri\":\"gs://***\"}",

so that the segment fetcher would know where to get the data from. Do you know if it's possible to do what I'm asking? Thanks

Mohit Garg4628

09/27/2022, 1:14 PM

Hi, We are using pinot latest version, we are trying to run query using V2 Engine. Getting following error [ { "message": "SQLParsingError\njava.lang.RuntimeException Error composing query plan for: select catalog_id from catalog_views_test\n\tat org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:131)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:147)\n\tat org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:125)\n\tat org.apache.pinot.broker.requesthandler.BrokerRequestHandler.handleRequest(BrokerRequestHandler.java:47)\n...\nCaused by: java.lang.NumberFormatException: null\n\tat java.base/java.lang.Integer.parseInt(Integer.java:614)\n\tat java.base/java.lang.Integer.parseInt(Integer.java:770)\n\tat org.apache.pinot.core.transport.ServerInstance.<init>(ServerInstance.java:63)\n\tat org.apache.pinot.query.routing.WorkerInstance.<init>(WorkerInstance.java:40)", "errorCode": 150 } ] Please help Thanks