Apache Pinot #troubleshooting

Deepak Mishra

03/30/2022, 6:12 AM

Hi Team , I am using pinot-0.10.0 to execute pinot ingestion job with avro file and it gives error like :- Caused by: java.lang.UnsupportedOperationException: Unsupported Avro type: NULL It is working fine with json file. can anyone please help into this issue?

Mathew Pallan

03/30/2022, 2:18 PM

Hi, I am evaluating Apache Pinot and wanted to understand the deep storage options while deploying on Azure. From the docs, it seems like Azure blob is not supported and Azure Data Lake Storage has to be used as the deep storage. Can you please confirm on the same. Also is PinotFS the abstraction used for deep storage as well. The docs mention PinotFS in the context of importing data and hence this query.

Elon

03/30/2022, 6:06 PM

Hi, if we make a change to the controller app, how do we build it? i.e.

npm install

and

npm run build

dmitry H

03/30/2022, 6:29 PM

Hi All, While deploying Pinot 0.9.3 in on-prem Tanzu K8 cluster I’m running into an issue where helm unable to pull zookeeper image from docker.io registry due to rate limiting limitation on docker.io. Normal Pulling 19s (x3 over 63s) kubelet Pulling image "zookeeper:3.5.5" Warning Failed 17s (x3 over 61s) kubelet Failed to pull image "zookeeper:3.5.5": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/zookeeper:3.5.5": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/zookeeper/manifests/sha256:b7a76ec06f68fd9c801b72dfd283701bc7d8a8b0609277a0d570e8e6768e4ad9: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Warning Failed 17s (x3 over 61s) kubelet Error: ErrImagePull Normal BackOff 3s (x3 over 60s) kubelet Back-off pulling image "zookeeper:3.5.5" Warning Failed 3s (x3 over 60s) kubelet Error: ImagePullBackOff To get around it I put zookeeper image into our internal image registry but not sure how to modify the helm chart to point it there. Can you assist in getting this resolved?

Daniel Lavoie

03/30/2022, 6:47 PM

Pinot is using a a child chart for zookeeper. You can configure the nested helm chart with

zookeeper.x

values

Luis Fernandez

03/30/2022, 7:03 PM

does anyone know why i may get the following error from the broker

"message": "2 servers [pinot-server-1_O, pinot-server-0_O] not responded"

I just changed the version of my broker trying to downgrade the entire system to

0.9.3

kaushal aggarwal

03/31/2022, 4:09 AM

can anyone help with this?

Arpita Bajpai

03/31/2022, 7:33 AM

Hi All, I am trying to read data from avro file to Pinot table using Batch Ingestion. I am facing error for STRING datatype . For now, I am providing null for the STRING field in avro file input and it is giving me below error: java.lang.RuntimeException: Caught exception while extracting data type from field:abcd Caused by: java.lang.UnsupportedOperationException: Unsupported Avro type: NULL In schema I have provided below config for the field : { "name": "abcd", "dataType": "STRING", "defaultNullValue": "none" }, I even tried without "defaultNullValue", for both am getting the same error mentioned above. I am passing value of field "abcd" as null in AVRO input file Can anyone help me with the same?

✅ 1

francoisa

03/31/2022, 12:54 PM

Hi 🙂 I’ve a misunderstanding on something that give me a lot of trouble. I’ve a neasted array on my JSON. I’ve managed to flaten it using the complexTypeConfig. Sound all good and generate me all row with correct values. But I’m tring to rename the genarated cols using eiter a groovy function or a JSON path string but none of them are working 😞 Ingestion.

Copy code

"ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "type",
          "transformFunction": "JSONPATHSTRING(data,'$.type')"
        }
      ],
      "complexTypeConfig": {
        "fieldsToUnnest": [
          "data.attributes.actualExpenses"
        ],
        "delimiter": "."
      }
    },

Schema is already defined with type STRING

Luis Fernandez

03/31/2022, 2:48 PM

pinot does not use spring right? asking because of this https://www.darkreading.com/application-security/zero-day-vulnerability-discovered-in-java-spring-framework

Diana Arnos

03/31/2022, 4:53 PM

The broker is failing to find the available servers for a lot of segments. This is the log message:

Copy code

Failed to find servers hosting segment: <segment> for table: <tableName>_REALTIME (all ONLINE/CONSUMING instances: [] and OFFLINE instances: [] are disabled, counting segment as unavailable)

How can I: 1- Make the brokers find the segments? 2- If 1 is not possible, how can I make the servers download or fetch all the missing segments?

abhinav wagle

03/31/2022, 10:28 PM

Trying to run the QueryRunner class with following command, and am seeing.

Copy code

java -jar pinot-tool-launcher-jar-with-dependencies.jar QueryRunner -mode singleThread -queryFile test.q -numTimesToRunQueries 0 -numIntervalsToReportAndClearStatistics 5 -brokerHost <host-name>

Grace Lu

04/01/2022, 2:48 AM

Hi team, if each of our pinot components has multiple resolvable dns name, I wonder if there is any recommended config for us to define/pick the hostname

Shailee Mehta

04/01/2022, 5:23 AM

Hello there, I am trying to validate this ~~schema~~ tableConfig using the schema validate API. It keeps returning 404 with reason null. Can someone help me find the issue with this schema

Copy code

{
  "tableName": "lineorder_star_OFFLINE",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "timeColumnName": "LO_ORDERDATE", //date field with day-granularity
    "timeType": "DAYS",
    "replication": "1",
    "schemaName": "lineorder"
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "metadata": {
    "customConfigs": {}
  },
  "tableIndexConfig": {
    "starTreeIndexConfigs": [
      {
        "dimensionsSplitOrder": [
          "LO_ORDERDATE", //date
          "LO_SUPPKEY", // dim field
          "LO_PARTKEY", // dim field
          "LO_DISCOUNT", // measure
          "LO_QUANTITY", //measure
          "LO_REVENUE", // dim field
          "LO_ORDERPRIORITY" //measure
        ],
        "skipStarNodeCreationForDimensions": [],
        "functionColumnPairs": [
          "SUM__LO_QUANTITY",
          "COUNT__LO_ORDERKEY",
          "SUM__LO_REVENUE"
        ]
      }
    ]
  }
}

Diogo Baeder

04/01/2022, 9:31 PM

Hi guys! Got a question: is it possible to, somehow, have a batch ingestion pipeline for daily ingestions (therefore meaning daily segments being created), but then, on a monthly basis, combine all segments for the previous month and delete the daily segments for it? I'll continue in this thread.

Diogo Baeder

04/01/2022, 9:42 PM

Another question, somewhat related to the previous one: is it possible to have a date column? And to set it up as the

timeColumn

? I was looking at using

DAYS

as the type for this, but it doesn't seem correct - I want to use real dates and not "days since Epoch".

ahsen m

04/02/2022, 1:47 AM

so i am getting following error any idea’s? zookeeper is running fine and kafka is using it. i am using existing zookeeper connection url in pinot.

Copy code

Opening socket connection to server kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181. Will not attempt to authenticate using SASL (unknown error)
53
Socket connection established, initiating session, client: /10.48.12.58:54236, server: kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181
52
Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
51
Opening socket connection to server kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181. Will not attempt to authenticate using SASL (unknown error)
50
Socket connection established, initiating session, client: /10.48.12.58:54248, server: kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181
49
Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
48
Opening socket connection to server kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181. Will not attempt to authenticate using SASL (unknown error)
47
Socket connection established, initiating session, client: /10.48.12.58:54268, server: kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181
46
Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
45
Opening socket connection to server kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181. Will not attempt to authenticate using SASL (unknown error)
44
Socket connection established, initiating session, client: /10.48.12.58:54274, server: kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181
43
Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
42
Opening socket connection to server kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181. Will not attempt to authenticate using SASL (unknown error)
41
Socket connection established, initiating session, client: /10.48.12.58:54278, server: kafka-cluster-zookeeper-client.kafka-cluster.svc.cluster.local/10.52.8.240:2181
40
Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
39
Failed to initialize Pinot Broker Starter
38
java.lang.NullPointerException: null
37
	at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
36
	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
35
	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
34
	at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1620) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
33
	at org.apache.helix.manager.zk.zookeeper.ZkClient.<init>(ZkClient.java:186) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
32
	at org.apache.helix.manager.zk.ZkClient.<init>(ZkClient.java:87) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
31
	at org.apache.helix.manager.zk.client.ZkConnectionManager.<init>(ZkConnectionManager.java:41) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
30
	at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
29
	at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
28
	at org.apache.pinot.common.utils.ServiceStartableUtils.applyClusterConfig(ServiceStartableUtils.java:54) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
27
	at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.init(BaseBrokerStarter.java:118) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
26
	at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:137) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
25
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
24
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:275) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
23
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:301) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
22
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:275) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
21
Failed to start a Pinot [BROKER] at 31.159 since launch
20
java.lang.NullPointerException: null
19
	at org.apache.helix.manager.zk.client.ZkConnectionManager.cleanupInactiveWatchers(ZkConnectionManager.java:112) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
18
	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:95) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
17
	at org.apache.helix.manager.zk.client.ZkConnectionManager.close(ZkConnectionManager.java:91) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
16
	at org.apache.helix.manager.zk.zookeeper.ZkClient.connect(ZkClient.java:1620) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
15
	at org.apache.helix.manager.zk.zookeeper.ZkClient.<init>(ZkClient.java:186) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
14
	at org.apache.helix.manager.zk.ZkClient.<init>(ZkClient.java:87) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
13
	at org.apache.helix.manager.zk.client.ZkConnectionManager.<init>(ZkConnectionManager.java:41) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
12
	at org.apache.helix.manager.zk.client.SharedZkClientFactory.getOrCreateZkConnectionNamanger(SharedZkClientFactory.java:60) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
11
	at org.apache.helix.manager.zk.client.SharedZkClientFactory.buildZkClient(SharedZkClientFactory.java:40) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
10
	at org.apache.pinot.common.utils.ServiceStartableUtils.applyClusterConfig(ServiceStartableUtils.java:54) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
9
	at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.init(BaseBrokerStarter.java:118) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
8
	at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:137) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
7
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
6
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:275) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
5
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:301) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
4
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:275) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
3
Shutting down Pinot Service Manager with all running Pinot instances...
2
Shutting down Pinot Service Manager admin application...
1
Deregistering service status handler

Shailesh Jha

04/03/2022, 3:04 PM

Hi Team My Pinot containers are not staring up suddenly. can anyone assist on this??

Copy code

pinot-zookeeper-0    0/1     CrashLoopBackOff   241        16h
pinot-zookeeper-1    0/1     Running            210        14h
pinot-zookeeper-2    0/1     CrashLoopBackOff   6          12m

zk Pod describe snap: Thanks

Shailee Mehta

04/03/2022, 4:54 PM

Hello there, After ingesting a file into pinot, I am able to see all the segments, but in a bad state. There were no errors while ingesting the file. However, the table is still empty. I can see the segment in the controllerUI also, they are properly created in the output folder. I see this error in the zooKeeper browser>instances>server>errors

Copy code

{
  "id": "100056e21a30006__lineorder_star_OFFLINE",
  "simpleFields": {},
  "mapFields": {
    "HELIX_ERROR     20220402-222348.000619 STATE_TRANSITION 631528bd-ca2b-4b60-b66e-01cfa58a44ff": {
      "AdditionalInfo": "Exception while executing a state transition task lineorder_star_OFFLINE_1992-01-01_1998-08-02_1java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)\n\tat org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)\n\tat org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)\n\tat org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts\n\tat org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61)\n\tat org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:72)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:148)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:142)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:164)\n\tat org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:158)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:406)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:393)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:385)\n\tat org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:372)\n\tat org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:355)\n\tat org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:162)\n\t... 12 more\n",
      "Class": "class org.apache.helix.messaging.handling.HelixStateTransitionHandler",
      "MSG_ID": "4dab2e7f-f693-4459-9cd6-6e837fa32c5b",
      "Message state": "READ"
    },
    "HELIX_ERROR     20220402-222348.000679 STATE_TRANSITION bca29fad-4930-48f8-b626-3013f465b1c8": {
      "AdditionalInfo": "Message execution failed. msgId: 4dab2e7f-f693-4459-9cd6-6e837fa32c5b, errorMsg: java.lang.reflect.InvocationTargetException",
      "Class": "class org.apache.helix.messaging.handling.HelixStateTransitionHandler",
      "MSG_ID": "4dab2e7f-f693-4459-9cd6-6e837fa32c5b",
      "Message state": "READ"
    }
  },
  "listFields": {}
}

controller.log

Jonathan Meyer

04/04/2022, 8:08 AM

Hello all ^^ I was looking into Slack history, trying to find an answer to my question - couldn't seem to find any so here I go We're doing standalone (apache/pinot Docker image in a WF) batch integrations - and we're seeing queries hitting Pinot before integrated data is available ("stale data") My use case is that we're doing data integration, firing off a Kafka event (after the

pinot-admin

step is finished), then querying Pinot, that's where we're seeing stale data Is there any way to • Have

./bin/pinot-admin.sh LaunchDataIngestionJob

wait for the data to be fully query-able ? • Have Pinot somehow notify when data becomes fully query-able ? NOTE: Job type is

SegmentCreationAndTarPush

Alice

04/04/2022, 10:27 AM

Hi, team: ‘java.lang.IllegalArgumentException: must provide a password for admin’ error occurred when I use ‘./bin/pinot-admin.sh StartBroker -configFileName ./conf/pinot-broker-7011.conf’ to start Pinot broker. I have the same config as listed in the example. I have started Pinot controller according to the example config and it started successfully. Any idea what’s wrong the the broker config? pinot.broker.access.control.class=org.apache.pinot.broker.broker.BasicAuthAccessControlFactory pinot.broker.access.control.principals=admin,user pinot.broker.access.control.principals.admin.password=verysecret pinot.broker.access.control.principals.user.password=secret

Diana Arnos

04/05/2022, 7:31 AM

Hello 👋 I have 3 of my 4 servers stuck with this kind of message:

Copy code

Find unloaded segment: <tableName>__0__35__20220404T0749Z, table: <tableName>_REALTIME, expected: ONLINE, actual: CONSUMING
Sleeping 1 second waiting for all segments loaded for partial-upsert table: <tableName>_REALTIME

Which endpoint should I use to try to sort this out? The

reload

one does not work, for the segment is still consuming and the

reset

always fails, for it can't stop a consuming segment for some reason. Would it be okay to just delete this segment? Would the Controller know it needs to be consumed again?

Lars-Kristian Svenøy

04/05/2022, 10:46 AM

Hello everyone. Is there any way to specify that we do not want any indexes for a field? We are struggling with a very large text blob, which seems to be stored in the indexes folder on the servers. We want the data to only reside on our deep store, and not be stored on disk at all. I’ve tried adding the field to the noDictionaryColumns and setting the fieldConfig encodingType to RAW, but it still seems to be creating a forward index which is stored on disk. Any ideas?

Saumya Upadhyay

04/05/2022, 1:36 PM

hi everyone, pinot is showing strange behavior after adding second broker, it is skipping data 4 times out of 10. It is happening even from pinot query console. Server logs are not showing any errors and because we have 6,7 realtime tables so logs are very quickly filling up and not able to track particular message from logs. Is there any way we can define groupId for consumer in table config.

Eduardo Cusa

04/05/2022, 1:55 PM

Hello guys, we're using the

ingestFromFile

endpoint to ingest data, but after some minutes the table is empty again. Do we need to set up a persistence backend?

David Cyze

04/05/2022, 4:01 PM

I’m trying to get Pinot running on a linux VM, outside of docker, with the quick start “transcript” data. Then, I want to query the data using the presto connector Last week, the docs recommended version 0.9.3. There were two issues I needed to resolve to get this to work: • The

timestamp

column in the schema needed to be renamed (I chose

timestamparoo

), because presto queries interpreted

timestamp

as a casting function as opposed to a column • The

timeFieldSpec

field in the table schema needed to change to

dateTimeFieldSpec

After making this changes, I could ingest and query (mostly) fine The docs have since recommended changing to 0.10.0, which I have tried doing. However, now when I run

./bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile ~/pinot-tutorial/transcript/batch-job-spec.yml

, I get an exception related to the timestamp column:

Copy code

Exception while collecting stats for column:timestamparoo in row:{
  "fieldToValueMap" : {
    "studentID" : 200,
    "firstName" : "Lucy",
    "lastName" : "Smith",
    "score" : 3.8,
    "gender" : "Female",
    "subject" : "Maths",
    "timestamparoo" : null
  },
  "nullValueFields" : [ ]
}

or.collect(LongColumnPreIndexStatsCollector.java:50) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.segment.local.segment.creator.impl.stats.SegmentPreIndexStatsCollectorImpl.collectRow(SegmentPreIndexStatsCollectorImpl.java:96) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]

It seems Pinot isn’t parsing the values for this column from the CSV. Why would that be? (More supporting files in thread)

Daniel

04/05/2022, 6:58 PM

Hi. I'm experimenting with Pinot in Kubernetes and used the Pinot helm chart in the getting started guide. The problem I am facing is that upon real-time ingestion from my Kafka topic, the Pinot servers and table segments get into a bad state after ingesting a few million records. The Pinot server pods encounter a JRE fatal error and restart. The Broker reports

Failed to find servers hosting segment: <segmentName> for table: <tableName> (all ONLINE/CONSUMING instances: [] and OFFLINE instances: [] are disabled, counting segment as unavailable)

Could this be due to an under-provisioned Pinot cluster? Something else?

Pankaj Thakkar

04/05/2022, 7:01 PM

Hey I am observing a peculiar behavior in my pinot setup. We have 3 offline servers each with about 366 segments. When I run a single query; all the servers respond fairly quickly and results come back within 100ms or so for this query. When I run the same query in parallel say about 100 times; one of the servers shows large scheduling delay (as printed by the server logs) but the execution time is much smaller. Due to this the overall queries now takes really long upto 4 seconds or so. Has anyone seen this behavior? I have tried attaching profiler to see what is happening and it indicates that the server that is showing more scheduling delay is more busy and doing more work than others. I am not able to figure out why this is the case. Any insights?

Alice

04/06/2022, 5:42 AM

Hi, I’m still testing the authentication on Pinot 0.10. With the following config, admin can log in, but user failed with tip of invalid username/password. I’m sure I use the same password as in the config. Any idea of the reason? Besides, with or without this config, #controller.segment.fetcher.auth.token=Basic YWRtaW46dmVyeXNlY3JldA, admin can log in. So what is it set for?

Arpita Bajpai

04/06/2022, 7:33 AM

Hi All, we are trying to pull all distinct records from our Pinot table along with pagination, but are unable to do so . Below are the details of our use case : In our Pinot table we have a "field_a" and "field_b" and every value in "field_a" is associated with multiple values in "field_b" field_a field_b aa 12 aa 13 aa 13 bb 45 bb 67 bb 78 We want all the unique or distinct combinations of field A and field B. That would be : field_a field_b aa 12 aa 13 bb 45 bb 67 bb 78 Also we have to run our query in batch of 1000 records at a time which we are trying to achieve through pagination, but it is not giving distinct records every time. Below is the sample query select field _a, field_b from table_name group by field_a, field_b limit 0,1000. Note:I was going through the Pinot docs and I found out that pagination does not work on Group By queries. Any suggestions on how we can achieve above use case ?