https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • k

    Kishore G

    03/12/2019, 7:47 PM
    awesome. Great job.
    ➕ 1
  • r

    Ravi

    03/12/2019, 8:26 PM
    This is a big milestone for Pinot project. I am really proud and excited to be part of this project for the last 3 years. Congrats to everyone who worked and participated in making this release happen from the entire community
  • s

    saj1th

    03/12/2019, 11:13 PM
    happy to see https://engineering.linkedin.com/blog/2019/03/pinot-joins-apache-incubator
  • s

    saj1th

    03/12/2019, 11:13 PM
    QQ - does pinot has something like https://github.com/druid-io/tranquility ?
  • s

    saj1th

    03/12/2019, 11:14 PM
    if not, what would be the recommendation if someone wants to do near-realtime ingestion from spark ?
  • r

    Ravi

    03/12/2019, 11:14 PM
    @User @User ^^
  • a

    Ananth Packkildurai

    03/12/2019, 11:16 PM
    @User Pinot has a Kafka consumer implemented part of the pinot servers. It is more of a pull model than tranquility like push model. (In the sense Pinot pull data from Kafka where tranquility push data to Druid), So Pinot don't require tranquility like a bridge to ingest data.
    👍 2
  • s

    saj1th

    03/12/2019, 11:17 PM
    Let's say i want to do some aggregation and window operations on my data ( hence using spark in the mix ) - what is the best way to do real-time ingestion from spark structured streaming ?
  • s

    saj1th

    03/12/2019, 11:18 PM
    I know that i could push it to kafka again and have it persisted to Pinot
  • s

    saj1th

    03/12/2019, 11:19 PM
    but if i were to create a custom sink from Spark - could that be supported ?
  • s

    saj1th

    03/12/2019, 11:21 PM
    CreateSegmentCommand - is all pull based - is it ?
  • r

    Ravi

    03/12/2019, 11:21 PM
    Pinot server now supports multiple event listeners. You can add your own listener as well. @User can point you in the right direction
  • s

    saj1th

    03/12/2019, 11:22 PM
    that's great news!
  • n

    Neha Pawar

    03/12/2019, 11:29 PM
    Hi @User, check out this doc: https://pinot.readthedocs.io/en/latest/pluggable_streams.html# It might be what you're looking for
  • s

    saj1th

    03/12/2019, 11:36 PM
    looks like pluggable streams are all pull based ( from streams with concepts like partition, checkpointing etc ) - is that assumption correct ?
  • s

    saj1th

    03/12/2019, 11:39 PM
    I was looking for a near-realtime push system 😞
  • s

    saj1th

    03/12/2019, 11:39 PM
    Thanks a lot for open-sourcing it and thanks for answering my queries
  • r

    Ravi

    03/12/2019, 11:40 PM
    You can always push to Kafka like systems.
  • s

    saj1th

    03/12/2019, 11:44 PM
    Wondering how uber is pushing to pinot from flink
    👍 1
  • s

    Subbu Subramaniam

    03/12/2019, 11:47 PM
    @User yes, pinot realtime consumption works via a pull model. You can build classes for your own stream as long as they meet some criteria. See the pluggable streams doc.
  • s

    Subbu Subramaniam

    03/12/2019, 11:48 PM
    Meanwhile, if I understand tranquillity correctly, it (can) gather data from Kafka and batch it. Can also batch from spark I guess. You can then make a pinot segment and push it to the controller as data for your table.
  • s

    saj1th

    03/12/2019, 11:48 PM
    may be i'm missing something - don't see any streams between flink and pinot
  • s

    Subbu Subramaniam

    03/12/2019, 11:48 PM
    You can read the classes that pinot-hadoop uses to build segment
  • s

    Subbu Subramaniam

    03/12/2019, 11:48 PM
    Alternative is to push the data into Kafka and have pinot read it
  • s

    saj1th

    03/12/2019, 11:48 PM
    may be they are creating segments and pushing it
  • s

    Subbu Subramaniam

    03/12/2019, 11:49 PM
    Sorry I was respnding to your earlier q of puill model
  • s

    Subbu Subramaniam

    03/12/2019, 11:49 PM
    I will let the uber guys respond to how they bridge flink
  • s

    saj1th

    03/12/2019, 11:51 PM
    thanks subbu . appreciate taking your time to answer the queries!
  • s

    Subbu Subramaniam

    03/12/2019, 11:52 PM
    anytime
  • k

    Kishore G

    03/13/2019, 12:12 AM
    @User Regarding you question on how Uber Pinot Workflow - Flink writes to Kafka and Pinot consumes from Kafka.
    👍 2
1...717273...160Latest