Apache Pinot #general

Join Slack

Kishore G

09/20/2019, 11:55 PM

Yes. It’s possible but needs some tuning on stream side and Pinot side

Andrew Greene

09/20/2019, 11:59 PM

Okay, thank you. 😃

Andrew Greene

09/20/2019, 11:59 PM

I'll give it a shot and get back to you.

Kishore G

09/24/2019, 2:53 AM

I am talking to Thomas later today at 8:30 pm. if anyone is interested in joining the call, let me know. They are trying to setup Pinot on Azure and facing some issues.

Kishore G

09/25/2019, 4:01 PM

@User Do we have instructions on how to configure PinotFS for ADSL?

Jennifer Dai

09/25/2019, 4:22 PM

yes, let me find the wiki

Jennifer Dai

09/25/2019, 4:23 PM

wanted to clarify - you mean ADL/azure data lake and not azure blob storage, right?

Jennifer Dai

09/25/2019, 4:24 PM

https://pinot.readthedocs.io/en/latest/pluggable_storage.html?highlight=pinotfs

Jennifer Dai

09/25/2019, 4:25 PM

this is the wiki, let me know if it's unclear

Mayank

09/25/2019, 4:25 PM

Thanks @User. Would be good to add a sample config in the doc.

Jennifer Dai

09/25/2019, 4:25 PM

do you mean for client secret/client id, etc?

Mayank

09/25/2019, 4:28 PM

Yes, a json sample of the config to provide.

Kishore G

09/25/2019, 4:53 PM

A sample config along segment upload command assuming data will help a lot

➕ 1

Jennifer Dai

09/25/2019, 5:05 PM

@User can you rephrase? do you mean http segment upload command?

Kishore G

09/25/2019, 5:07 PM

Yes

Kishore G

09/25/2019, 5:08 PM

A sample curl call

vibhor

09/27/2019, 5:40 AM

my pinot setup is able to scan only 7 segments out of many segments..

vibhor

09/27/2019, 5:40 AM

can someone help on this?

vibhor

09/27/2019, 5:45 AM

<!here>

Kishore G

09/27/2019, 5:46 AM

Sure

Kishore G

09/27/2019, 5:46 AM

How many segments do you have

Kishore G

09/27/2019, 5:47 AM

Can you paste the query and response

vibhor

09/27/2019, 5:50 AM

{"pql":"select count(*) from dauTable"} .

vibhor

09/27/2019, 5:50 AM

Copy code

{
  "aggregationResults": [
    {
      "function": "count_star",
      "value": "12488040"
    }
  ],
  "exceptions": [],
  "numServersQueried": 1,
  "numServersResponded": 1,
  "numSegmentsQueried": 7,
  "numSegmentsProcessed": 7,
  "numSegmentsMatched": 7,
  "numDocsScanned": 12488040,
  "numEntriesScannedInFilter": 0,
  "numEntriesScannedPostFilter": 0,
  "numGroupsLimitReached": false,
  "totalDocs": 12488040,
  "timeUsedMs": 4,
  "segmentStatistics": [],
  "traceInfo": {}
}

vibhor

09/27/2019, 5:50 AM

no of segments : 600

Kishore G

09/27/2019, 5:52 AM

How did you get the number of segments

Kishore G

09/27/2019, 5:52 AM

From the response it looks like it queried all the segments that are ONLINE

vibhor

09/27/2019, 5:55 AM

outdir has 600 folders with name dauSegments_600 (e.g) ..

vibhor

09/27/2019, 5:55 AM

and we have data mismatch for the rows hence

Kishore G

09/27/2019, 5:56 AM

Maybe all the segments were not uploaded?