Apache Pinot #troubleshooting

Anish Nair

12/30/2021, 8:00 AM

Hi Team, We have set following flush config for realtime table "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "12h", "realtime.segment.flush.threshold.segment.size": "200M", "realtime.segment.flush.autotune.initialRows": "10000000", But still seeing segments getting generated with size less than 50mb (attaching screenshot for reference ) Can someone help?

Syed Akram

01/03/2022, 8:49 AM

Hi, I have 2 pinot nodes (where server+controller+broker) running in each node. suppose node1 is down for some reason and i ingested data with replication 2 , where data will be ingested to node2 only... in this scenario, if i query from node2, data is not getting served... is this expected? after some i made node1 up.... then only data is queryable. can someone explain this?

eywek

01/03/2022, 2:46 PM

Hello, do you have any news on this issue https://github.com/apache/pinot/issues/7461 ? I have a column named

timestamp

and I’m unable to use PrestoDB since double quotes are removed

Yeongju Kang

01/04/2022, 5:43 AM

Hi team, I am testing hybrid table feature but met 2 problems and summarized what I met. 1. Does broker consider both time column and primary key? The value of user_id was same as '1' but time columns' value were different and gave me result as below.

Copy code

select * from user123 where user_id = '1' -- returned 2 rows
select * from user123_REALTIME where user_id = '1' -- returned 1 row
select * from user123_OFFLINE where user_id = '1' -- returned 1 row

2. Inconsistent results are returned for realtime table. I met this when I was validating above. I configured replicasPerPartition --> 2 stream.kafka.broker.list --> 3 brokers' endpoints stream.kafka.consumer.type --> lowLevel

Copy code

select user_id, count(*) from user123
group by user_id
-- case 4 servers were queried : returned 102 rows
-- case 3 servers were queried : returned 96 rows

Vibhor Jaiswal

01/05/2022, 11:53 PM

Hi All .. While querying the Pinot from Trino i am getting "max split per segment is 50000" error .Tried set

Copy code

pinot.max-rows-per-split-for-segment-queries=2147483647

But trino says session property pinot.max-rows-per-split-for-segment-queries does not exist .Please suggest if you can . i was following this -https://www.mail-archive.com/dev@pinot.apache.org/msg02773.html

xtrntr

01/06/2022, 12:01 PM

i have my servers running on k8s helm with the following resources:

Copy code

# <https://docs.pinot.apache.org/operators/tutorials/deployment-pinot-on-kubernetes#jvm-setting>
resources:
  requests:
    cpu: 4
    memory: 10G
  limits:
    cpu: 4
    memory: 10G

how do i diagnose the cause of slow queries based on the logs?

Copy code

Processed requestId=93,table=eventsv2-2021-09_OFFLINE,segments(queried/processed/matched/consuming)=10/10/10/-1,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=4883,resSerMs=1,totalTimeMs=4885,minConsumingFreshnessMs=-1,broker=Broker_pinot-broker-0.pinot-broker-headless.pinot.svc.cluster.local_8099,numDocsScanned=1194957,scanInFilter=422644416,scanPostFilter=1194957,sched=fcfs,threadCpuTimeNs=0

...

Processed requestId=93,table=eventsv2-2021-09_OFFLINE,segments(queried/processed/matched/consuming)=10/10/10/-1,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=4414,resSerMs=0,totalTimeMs=4415,minConsumingFreshnessMs=-1,broker=Broker_pinot-broker-0.pinot-broker-headless.pinot.svc.cluster.local_8099,numDocsScanned=1156729,scanInFilter=422867040,scanPostFilter=1156729,sched=fcfs,threadCpuTimeNs=0

the offending query is:

Copy code

SELECT user, COUNT(*) FROM "eventsv2-2021-09" WHERE IN_SUBQUERY(cell, 'SELECT ID_SET(location) FROM dimTable WHERE id IN (...)') = 1 GROUP BY user HAVING COUNT(user) >= 20 LIMIT 10000000

xtrntr

01/06/2022, 12:04 PM

related question: is it possible to call

IN_ID_SET

inside

IN_SUBQUERY

? i’ve tried, but i seem to keep running into sql quoting issues:

Copy code

Was expecting one of:\n    \")\" ...\n    \",\" ...\n    <QUOTED_STRING> ...","errorCode":150

Tiger Zhao

01/06/2022, 5:43 PM

Hi, occasionally when calling delete schema through the rest API, I get a null pointer exception. Any ideas as to why this happens? (see logs in thread)

Weixiang Sun

01/06/2022, 5:57 PM

what is the default max length of String field?

Sadim Nadeem

01/07/2022, 5:49 AM

issue pinot.JPG

Arpita Bajpai

01/07/2022, 6:09 AM

Hi All, I am trying to connect to pinot using JDBC client. I am able to connect it locally, but when am trying to access the URL with Basic Auth it is throwing "PinotClientException: Pinot returned HTTP status 403, expected 200" . I did provide username and password using setProperties method but still the same . Can anyone help with the same ?

Elon

01/07/2022, 9:45 PM

We noticed that some brokers are missing routing table entries for segments. This results in different query results depending on which broker is used. After restarting the broker multiple times it picks up the segments (which are not newly added - a lot of them are 2+ days old). Here is the error message:

Copy code

2022/01/07 11:39:58.204 WARN [BaseInstanceSelector] [HelixTaskExecutor-message_handle_thread] Failed to find servers hosting segment: MYSECRETTABLE-1641542240329_2022-01-06_2022-01-06_6 for table: MYSECRETTABLE

Is there any way to reduce/eliminate these errors? We're looking at zk client configuration in helix...

➕ 1

Mayank

01/09/2022, 3:53 PM

You have 1:HOURS in there?

Mayank

01/09/2022, 5:21 PM

Seems like a new check added: @Rong R ^^

👀 1

Lars-Kristian Svenøy

01/10/2022, 11:37 AM

Hello everyone 👋 I’m currently running the Pinot Recommendation engine, and have a question regarding the output I’m seeing. One of the kafka topics we have has 100,000 messages/second, and Pinot is recommending 400 partitions for this topic.. I can see in the code it simply takes the number of messages and divides it by 250. Is 400 partitions really recommended here? Or should I disregard it.. Not sure what the impact is either way

xtrntr

01/10/2022, 12:12 PM

https://docs.pinot.apache.org/basics/indexing/forward-index if i want to sort a column, should i sort within each partition (parquet file) or have it globally sorted across all partitions?

Mark Needham

01/10/2022, 1:36 PM

I'm not 100% sure those docs are right - I think you do need to explicitly specify the

sortedColumn

in your table config. Or have you already tried setting it?

Lars-Kristian Svenøy

01/10/2022, 8:23 PM

Hello again 👋 Again thanks for all the help. I have a question regarding the RealtimeToOfflineSegmentsTask.. If I change my instanceAssignmentConfigMap to move completed segments to my offline tenants, will they still be picked up by the task later down the line? Or does changing the tag essentially move it to the offline table, meaning it won’t be moved with the task?

abhinav wagle

01/10/2022, 10:37 PM

Hello trying to write a batch Job ingestion spec for CSVRecordReader. Where do I specify the delimiter config for CSVRecordReader ? https://docs.pinot.apache.org/configuration-reference/job-specification#record-reader-spec Field delimiter is

\t

in my case

Anish Nair

01/11/2022, 7:52 AM

Hi Team, Regarding indexes, in Realtime Table config we have provided the following:

Copy code

"tableIndexConfig": {
      "invertedIndexColumns": [
        "advertiser_id",
        "partner_id",
        "market_place_id"
      ],
"rangeIndexVersion": 1,
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"loadMode": "MMAP",
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": true

When i check the table metadata, i get : http://serve1:9000/segments/reporting_aggregations/metadata?type=REALTIME

Copy code

{
  "reporting_aggregations__0__0__20220111T0608Z": {
    "segmentName": "reporting_aggregations__0__0__20220111T0608Z",
    "schemaName": null,
    "crc": 1605865976,
    "creationTimeMillis": 1641883074848,
    "creationTimeReadable": "2022-01-11T06:37:54:848 UTC",
    "timeGranularitySec": 0,
    "startTimeMillis": 1641790800000,
    "startTimeReadable": "2022-01-10T05:00:00.000Z",
    "endTimeMillis": 1641859200000,
    "endTimeReadable": "2022-01-11T00:00:00.000Z",
    "segmentVersion": "v3",
    "creatorName": null,
    "custom": {},
    "columns": [],
    "indexes": {},
    "star-tree-index": null
  }
}

and also when checked /pinot/data/server/index/reporting_aggregations_REALTIME/reporting_aggregations__0__0__20220111T0608Z/v3/metadata.properties , following is visible for each and every column in table

Copy code

column.postal_code.hasDictionary = true
column.postal_code.hasInvertedIndex = true

How to check which is correct? am i missing something in config?

Aditya

01/11/2022, 8:57 AM

Hi Folks I am facing an issue with sorted index on an offline table. The sortedColumn is set to user_id (String) and input csv file is physically sorted on this column.

Copy code

"sortedColumn": [
        "user_id"
      ]

Metadata shows the column is sorted if row count in segment is < 2.5M (~~180 MB), if row count is increased to 3M (~~ 220 MB) the metadata shows isSorted value as false Using the

org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner

and

jobType: SegmentCreationAndTarPush

for creating and pushing the segments. The segment creation job logs shows its creating dictionary (default) for each column. Is there a limit to number of rows/segment size if we use sorted index? Could this be related to cardinality of column user_id?

Lars-Kristian Svenøy

01/11/2022, 3:08 PM

Hey everyone. I’m running the RealtimeToOfflineSegmentsTask on some large tables, and unfortunately some of these tasks take longer than 1 hour to complete, triggering the timeout. Is the minion task timeout configurable?

Elon

01/11/2022, 11:51 PM

Is there any issue with having a large number of tables on a single cluster in terms of zk latency, metadata size, etc.? We currently have ~75 tables.

Sadim Nadeem

01/12/2022, 7:16 AM

what are some ways can we ensure High Availability / Disaster Recovery / Failover of my pinot cluster running on google kubernetes engine .. • Any advice will be very helpful to ensure almost 100% up time.. • We have alerts configured for CPU/ram/disk on slack .. • We have logging configured on loki • we have 3 broker , controller , server, zookeeper pods each with enough CPU, ram and disk as per load(incoming feeds frequency on kafka topics) .. • we have only Realtime tables Note: we encountered a crash after 32 days most probably because of hard disk exhausted and because of restarting zookeeper pods using kubectl delete while debugging.. cluster became unstable and attached persistent volume claims data got lost .. and we had data loss.. cc: @Mayank @Xiang Fu @Subbu Subramaniam @Jackie

Lars-Kristian Svenøy

01/12/2022, 12:56 PM

Hey everyone 👋 I’m experiencing an issue which makes it seem like the RealtimeToOfflineSegmentsTask is breaking my segments. Just wanted to check here before I raise an issue.. The minion task to convert the segments is taking a very long time, which causes the task to timeout. But while it has timed out, it looks like it still partially completes, and deleting segments and replacing them with segments in deep store with a strange prefix. Example segment: table_name__11__312__20220111T0936Za96a67f8-23e8-4213-b89b-9b199f3ffb21 In the actual table, all segments which end up like this show up as bad in error state, and can not be reloaded. Any guidance?

xtrntr

01/13/2022, 4:41 AM

sorry, whats the difference? edit: also tried setting

pinot.server.query.executor.num.groups.limit=-1

Prashant Pandey

01/13/2022, 2:59 PM

Hi Pinot folks, we had a linear increase in consumer lag in a single partition in one of our topics (all other partitions were fine). We want to find out which realtime server is consuming from this particular partition. Is there a way to know this segment assignment? Thanks!

Mark Needham

01/13/2022, 5:42 PM

controller.port-9000

change that to:

controller.port=9000

abhinav wagle

01/14/2022, 2:54 AM

Copy code

java.lang.NoSuchMethodException: org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main([Ljava.lang.String;)
	at java.lang.Class.getMethod(Class.java:1786)
	at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:669)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:461)
	at <http://org.apache.spark.deploy.yarn.ApplicationMaster.org|org.apache.spark.deploy.yarn.ApplicationMaster.org>$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:773)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
	at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:772)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:797)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
22/01/13 23:40:15 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.lang.NoSuchMethodException: org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main([Ljava.lang.String;)
	at java.lang.Class.getMethod(Class.java:1786)
	at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:669)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:461)
	at <http://org.apache.spark.deploy.yarn.ApplicationMaster.org|org.apache.spark.deploy.yarn.ApplicationMaster.org>$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:773)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
	at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:772)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:797)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

abhinav wagle

01/14/2022, 6:59 PM

Hellos, See anything odd in below command to which launches

LaunchDataIngestionJobCommand

. It keeps picking

StartKafkaCommand

execution even if I have

LaunchDataIngestionJob

passed as parameter. Running on spark

2.4.0

Pinot :

0.10.0

Copy code

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-cores 4 \
  --executor-cores 4 \
  --driver-memory 4g \
  --executor-memory 4g \
  --class org.apache.pinot.tools.admin.PinotAdministrator /tmp/awagle/pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar LaunchDataIngestionJob -jobSpecFile /tmp/awagle/sparkIngestionJobSpec.yaml