@Mayank @Xiang Fu @Jackie@Kishore G @Daniel Lavoie @Ken Krugler @Neha Pawar.. we are actually trying upsert above here as mentioned by @Radhika...but the table count is coming up as zero .. we have followed all the steps required as mentioned here https://docs.pinot.apache.org/basics/data-import/upsert .. we are using Apache samza API(check attached code snippet) for partition by as mentioned in the above doc for:- Partition the input stream by the primary key An important requirement for the Pinot upsert table is to partition the input stream by the primary key. For Kafka messages, this means the producer shall set the key in the

send

API. If the original stream is not partitioned, then a streaming processing job (e.g. Flink) is needed to shuffle and repartition the input stream into a partitioned one for Pinot's ingestion.

we can see this streaming application publishing data on the output topic on top of which this pinot upsert table is created and before writing the data . using samza API .. we are shuffling the data to push data with same key on same partition .. still the data is not ingested by pinot and table count comes as zero . the table schema and table creation script are shared above by @Radhika

Hi guys!

I have a question regarding Kafka integration with Pinot

Thanks in advance!

Hi everyone, good morning 🙂. I need some help debugging slow queries in our Pinot cluster. We are running the following query:

Select api_id, service_name, service_id, api_name, COUNT(*) FROM myTable WHERE tenant_id = 'someTenantId' AND ( api_id IS NOT NULL AND start_time_millis >= 1625039026768 AND start_time_millis < 1625643826768 ) GROUP BY api_id, service_name, service_id, api_name ORDER BY PERCENTILETDIGEST99(duration_millis) desc  limit 10000

And these are the query stats:

timeUsedMs: 1077
numDocsScanned: 560325713
totalDocs: 3103044892
numServersQueried: 8
numServersResponded: 8
numSegmentsQueried: 623
numSegmentsProcessed: 115
numSegmentsMatched: 115
numConsumingSegmentsQueried: 4
numEntriesScannedInFilter: 25000000
numEntriesScannedPostFilter: 2801628565
numGroupsLimitReached: false
partialResponse: -
minConsumingFreshnessTimeMs: 1626154723247

The most conspicuous of these stats is

numEntriesScannedInFilter

. The troubleshooting guide says that if this number is too high, we should consider adding an index on the column, While we don’t have an index on this, our segment config is:

"segmentsConfig": {
      "timeType": "MILLISECONDS",
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "timeColumnName": "start_time_millis",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "7",
      "replicasPerPartition": "1",
      "schemaName": "rawServiceView"
    }

As you can see, the

timeColumnName

is

start_time_millis

and therefore, we haven’t added any index on this column (our reasoning is that segments would be pruned on this column anyway so we don’t need an extra index).

myTable

is a real-time table. If I remove the filter on

start_time_millis

, then

numEntriesScannedInFilter

becomes 0. What are we doing wrong here?

there is nothing wrong, its working as expected

a segment is either • no match • full match

• partial match

no match or full match will not add to numEntriesScannedInFilter

but the partial ones will have to scan to evaluate the time filter

if you want to bring this further down, you can try range index on time column

Thanks for the reply @Kishore G we’ll add an index on start_time_millis and report back.

it should be range index

Yes, adding a range index only @Kishore G. We already have inverted indices for the other two fields. Let me test it and report back.

So, um, the dependencies seem to be a wee bit out of date for some things. Hitting this attempting to run the spark job on EMR 6.3.0/jdk 11: https://issues.apache.org/jira/browse/LANG-1384

Caused by: java.lang.NullPointerException
        at org.apache.commons.lang3.SystemUtils.isJavaVersionAtLeast(SystemUtils.java:1626)
        at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
        at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)

hello

I am trying to follow the steps outlined on the link https://docs.pinot.apache.org/basics/data-import/batch-ingestion for setting up a schema and table data in Pinot

I am able to upload the schema and the table structure using the commands bin/pinot-admin.sh AddTable \\ -tableConfigFile /path/to/table-config.json \\ -schemaFile /path/to/table-schema.json -exec

But when I am trying to load the csv file data into the table using bin/pinot-admin.sh LaunchDataIngestionJob \\ -jobSpecFile /tmp/pinot-quick-start/batch-job-spec.yml

I am getting error as follows "Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0: <http://localhost:9000>/tables/transcript/schema"

I am unable to understand why - I am doing nothing special - just following the steps outlined in the document

Can anyone help me with this issue

I’m running Pinot on local Mac

Not on spark etc

that's why it was unable to locate the file and giving the error accordingly.

Pinot is amazing

Hello! I am trying to create or update a job spec over http. Is there any

LaunchDataIngestionJob

http endpoint to be called? I could not find in the swagger interface to upload the jobSpec yaml file.

@Luiz Gabriel Lima Pinheiro is this for production or poc?