Apache Pinot #troubleshooting

Neha Pawar

07/27/2020, 8:09 PM

is this table identical to the v1, other than name of table and timeColumnName? could you paste the schema too? and the whole stackstrace?

Neha Pawar

07/27/2020, 8:31 PM

@Elon ^^

Elon

07/27/2020, 8:35 PM

It is, let me paste the schema:

Neha Pawar

07/27/2020, 9:38 PM

@Xiang Fu @Daniel Lavoie ^^

Elon

07/28/2020, 5:50 AM

Looks like the https://docs.pinot.apache.org/user-guide/user-guide-query/pinot-query-language link is broken

Xiang Fu

07/28/2020, 7:28 AM

It should be

<https://docs.pinot.apache.org/users/user-guide-query/pinot-query-language>

now

Apoorva Moghey

07/29/2020, 9:00 AM

I just pulled the latest version of the pinot. Post that I am getting this error on starting the server. Caught exception while validating schema: wallet_success_rate java.lang.IllegalStateException: Incorrect granularity size: 1. Must be of format '[0-9]+:<TimeUnit>' This was working pre pull.

Apoorva Moghey

07/29/2020, 9:01 AM

Copy code

{
  "schemaName": "wallet_success_rate",
  "dimensionFieldSpecs": [
    {
      "name": "orderId",
      "dataType": "STRING",
      "singleValueField": true,
      "defaultNullValue": "null"
    },
    {
      "name": "status",
      "dataType": "STRING",
      "singleValueField": true,
      "defaultNullValue": "null"
    },
    {
      "name": "login",
      "dataType": "STRING",
      "singleValueField": true,
      "defaultNullValue": "null"
    },
    {
      "name": "paymentOptionType",
      "dataType": "STRING",
      "singleValueField": true,
      "defaultNullValue": "null"
    },
    {
      "name": "provider",
      "dataType": "STRING",
      "singleValueField": true,
      "defaultNullValue": "null"
    },
    {
      "name": "adtf",
      "dataType": "BOOLEAN",
      "singleValueField": true,
      "defaultNullValue": "false"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "amount",
      "dataType": "DOUBLE",
      "defaultNullValue": 0
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "createdAt",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    },
    {
      "name": "years",
      "dataType": "INT",
      "format": "1:MONTHS:SIMPLE_DATE_FORMAT:YYYY",
      "granularity": "1:YEARS"
    },
    {
      "name": "months",
      "dataType": "INT",
      "format": "1:MONTHS:SIMPLE_DATE_FORMAT:MM",
      "granularity": "1:MONTHS"
    },
    {
      "name": "days",
      "dataType": "INT",
      "format": "1:DAYS:SIMPLE_DATE_FORMAT:dd",
      "granularity": "1:DAYS"
    },
    {
      "name": "hours",
      "dataType": "INT",
      "format": "1:HOURS:SIMPLE_DATE_FORMAT:hh",
      "granularity": "1:HOURS"
    },
    {
      "name": "minutes",
      "dataType": "INT",
      "format": "1:MINUTES:SIMPLE_DATE_FORMAT:mm",
      "granularity": "1:MINUTES"
    }
  ]
}

Apoorva Moghey

07/29/2020, 9:01 AM

This is the schema I am using

Xiang Fu

07/29/2020, 10:49 AM

I guess pinot doesn’t support YEARS or MONTHS. Pinot uses java TimeUnit which supports up to DAYS granularity

Apoorva Moghey

07/29/2020, 11:01 AM

Yeah just got it... After checking

DateTimeGranularitySpec

Apoorva Moghey

07/29/2020, 11:02 AM

Moving it to dimensions

Oguzhan Mangir

07/31/2020, 9:15 AM

can someone help me about maven enforcer rules? 😞

Yash Agarwal

07/31/2020, 10:52 AM

I am getting the following exception when querying the controller with a specific sql query.

Copy code

Caught exception while sending query request
java.io.IOException: Failed : HTTP error code : 500
	at org.apache.pinot.controller.api.resources.PinotQueryResource.sendPostRaw(PinotQueryResource.java:284) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
	at org.apache.pinot.controller.api.resources.PinotQueryResource.sendRequestRaw(PinotQueryResource.java:322) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
	at org.apache.pinot.controller.api.resources.PinotQueryResource.getQueryResponse(PinotQueryResource.java:204) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]
	at org.apache.pinot.controller.api.resources.PinotQueryResource.handlePostSql(PinotQueryResource.java:131) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d177f0]

If I directly query the broker I am getting

Copy code

<html>

<head>
	<title>Grizzly 2.4.4</title>
	<style>
		<!--div.header {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#003300;font-size:22px;-moz-border-radius-topleft: 10px;border-top-left-radius: 10px;-moz-border-radius-topright: 10px;border-top-right-radius: 10px;padding-left: 5px}div.body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:#FFFFCC;font-size:16px;padding-top:10px;padding-bottom:10px;padding-left:10px}div.footer {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#666633;font-size:14px;-moz-border-radius-bottomleft: 10px;border-bottom-left-radius: 10px;-moz-border-radius-bottomright: 10px;border-bottom-right-radius: 10px;padding-left: 5px}BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}B {font-family:Tahoma,Arial,sans-serif;color:black;}A {color : black;}HR {color : #999966;}
		-->
	</style>
</head>

<body>
	<div class="header">Request failed.</div>
	<div class="body">Request failed.</div>
	<div class="footer">Grizzly 2.4.4</div>
</body>

</html>

I have recently added a config in broker

Copy code

pinot.broker.timeoutMs=120000

Any Ideas ?

Yash Agarwal

07/31/2020, 10:54 AM

There are no logs in broker. Other queries are working fine, this query would scan 1.7 Bil recods and do a sum of 3 metrics and distinctcount of 1 dimension

Yash Agarwal

07/31/2020, 12:25 PM

It might be because of high memory usage on broker. as i am seeing usage spikes upto 60 - 80 %

Mayank

07/31/2020, 2:08 PM

Yes, distinctcount on 1.7B records may the likely issue. Have you considered using HLL based approximate distinctcount?

Yash Agarwal

07/31/2020, 2:09 PM

We have tried HLL in the past. We are not looking for approximations.

Yash Agarwal

07/31/2020, 2:13 PM

Also seems like that is the problem, I tried without the distinctcount and it is returning successfully in 2 secs.

Buchi Reddy

07/31/2020, 4:20 PM

@Buchi Reddy has left the channel

Yash Agarwal

08/03/2020, 8:49 AM

Is it necessary to move my segments to controller and use that as the download uri? I have an hdfs deep storage and want to continue using that to store segments.

Yash Agarwal

08/03/2020, 8:52 AM

Copy code

executionFrameworkSpec:
  name: 'spark'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentUriPushJobRunner'
  extraConfigs:
    stagingDir: <hdfs://hdfs_base/staging_dir>

jobType: SegmentUriPush
inputDirURI: '<hdfs://hdfs_base/raw_data_dir>'
includeFileNamePattern: 'glob:**/*.orc'
outputDirURI: '<hdfs://hdfs_base/segment_dir>'
overwriteOutput: true

pinotFSSpecs:
  - scheme: hdfs
    className: org.apache.pinot.plugin.filesystem.HadoopPinotFS
    configs:
      'hadoop.conf.path': '/etc/hadoop/conf/'

recordReaderSpec:
  dataFormat: 'orc'
  className: 'org.apache.pinot.plugin.inputformat.orc.ORCRecordReader'
  
tableSpec:
  tableName: 'tableName'
  schemaURI: '<http://controller:9000/tables/tableName/schema>'
  tableConfigURI: '<http://controller:9000/tables/tableName>'

segmentNameGeneratorSpec:
  type: simple
  configs:
    segment.name.postfix: 'batch'

pinotClusterSpecs:
  - controllerURI: '<http://controller:9000>'

pushJobSpec:
  pushParallelism: 50
  pushAttempts: 2
  pushRetryIntervalMillis: 1000
  segmentUriPrefix: <https://file_server/fs/hdfs_base>
  segmentUriSuffix: ''

Yash Agarwal

08/03/2020, 8:53 AM

but this is downloading all the segments in controller and using that path to set in

segment.offline.download.url

Kishore G

08/03/2020, 1:31 PM

What is the dataDir and day setting setting in controller

Yash Agarwal

08/03/2020, 2:01 PM

For dataDir I use the following path.

Copy code

-dataDir /var/pinot/controller/data

I didn’t get the question about the day setting ?

Yash Agarwal

08/03/2020, 2:06 PM

Controller is using a local file system to store temp segments.

Yash Agarwal

08/03/2020, 2:22 PM

Copy code

// This boolean is here for V1 segment upload, where we keep the segment in the downloadURI sent in the header.
// We will deprecate this behavior eventually.
if (!moveSegmentToFinalLocation)

Yash Agarwal

08/03/2020, 2:22 PM

I think it is because of this.

Mayank

08/03/2020, 3:34 PM

I think you need change the dataDir to be an hdfs:// uri

Yash Agarwal

08/03/2020, 3:53 PM

I cannot connect directly to hdfs. We only have access through an http api.