I’ve filed a bunch of issues I’ve encountered recently

Hello

I don’t know if the latter 2 have been solved by this https://github.com/apache/incubator-pinot/pull/5213

is there thirdeye channel ?

@User apache-thirdeye.slack.com

Hey all, past few months have been very exciting working on pinot. The journey that @User and I went through and the learnings that we have gained from the pinot community, we have tried to project on the blog. This would not have been possible without the great support that you all provided, especially @User @User @User @User @User. Please share it across😁 https://www.linkedin.com/posts/shounak-kulkarni-22ba30169_tasted-apache-pinot-and-we-loved-it-activity-6689151508224864256-a6Zt

Awesome post @User. Thanks for sharing. Being able to ingest 120k events/sec on a single node is indeed insane!

Nice post, @User

Great blog as well as great work squeezing insane performance from tiny footprint @User @User

we're planning to try out pinot and your blog post made us more excited about it @User. thanks for detailed write up!

Glad that the blog was helpful! @User

<!here> i am currently working on a Kafka connectors (source and sink). The sink is more important I would say. I spent the last two weeks digging into the codebase. I noticed within org.apache.pinot.spi.stream package there are a number of classes/interfaces that are implementing the consumption from Kafka if i am not wrong and within pinot-core -> org.apache.pinot.core.realtime

I don’t want to reinvent the while here: but my goal is to be able to do the same ( consumer from Kafka using Kafka Connect API -> push/sink the data into pinot

now i stated my goal

here is a few screen shots of what i built so far

i will open source the code this week

Screen Shot 2020-07-20 at 2.53.29 PM.png

Screen Shot 2020-07-20 at 2.54.19 PM.png

Screen Shot 2020-07-20 at 2.54.34 PM.png

Screen Shot 2020-07-20 at 2.54.49 PM.png

@User this is pretty cool, can we do this in batches to start with?

Yes, that is fine, I just need collaborate with someone in your team at the beginning at least

@User @User built something similar to this in Flink to generate the segment and push to Pinot periodically

lets create a kafka-connect channel and move this discussion to that channel. Super excited about this!

Awesome

Quick question about Pinot query. If this was the Pinot query result plan, does this mean

numSegmentsProcessed

segments were mem mapped?

{
  "resultTable": {
    "dataSchema": {
      "columnDataTypes": [
        "BYTES"
      ],
      "columnNames": [
        "id"
      ]
    },
    "rows": [
      [
        "6a254bd3c853e950"
      ]
    ]
  },
  "exceptions": [],
  "numServersQueried": 4,
  "numServersResponded": 4,
  "numSegmentsQueried": 1237,
  "numSegmentsProcessed": 1229,
  "numSegmentsMatched": 4,
  "numConsumingSegmentsQueried": 8,
  "numDocsScanned": 4,
  "numEntriesScannedInFilter": 4,
  "numEntriesScannedPostFilter": 4,
  "numGroupsLimitReached": false,
  "totalDocs": 265510367,
  "timeUsedMs": 32,
  "segmentStatistics": [],
  "traceInfo": {},
  "minConsumingFreshnessTimeMs": 1595297570989
}

Wouldn't Pinot make an incredible datasource for Grafana?