Apache Pinot #general

Karin Wolok

03/29/2021, 8:11 PM

👋 Hello new Pinot members! 🍷 Who are you? What brought you here? Please introduce yourself to the community, we'd love to learn about you! 🙂 @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

🙌 2

Ralph Debusmann

03/29/2021, 8:27 PM

Hi brothers, I am from Bosch Power Tools in Germany and I am interested in using Pinot to replace some of our Kafka Streams apps to power Industrial IoT and other Big Data use cases…

🍷 5

👋 7

Karin Wolok

04/07/2021, 6:03 PM

Hey ya'll! 📣 🎉 We're working on organizing a series of meetup events starting in May on for a couple of months. 📅 👀 We're looking for presenters on use cases, how-to's, your experiences working with Pinot, "getting started with X in Pinot", features and connectors. There's really no limit with types of topics. It can almost be looked at as like a show and tell. Even if your use case isn't fully built out, many people might be interested to see what are you working on, what made you think of Pinot, what were you doing before, what led you here, what works for you and what doesn't, how you compared your options, comparisons of Pinot and other solutions, etc. !! Anything that could help newbies and seasoned developers is a great thing! There's some talks at Kafka summit that might add some inspiration. https://www.kafka-summit.org/events/kafka-summit-europe-2021/agenda If you're interested, feel free to DM me directly for questions or to share thoughts / ideas. You can also submit your an idea here: https://bit.ly/pinotspeaker The sooner you submit. the better. We will start scheduling these top of next week!

👍 3

👀 4

🍷 5

📢 1

Ravishankar Nair

04/10/2021, 6:46 AM

Joined Pinot Slack Channel....

Kishore G

04/13/2021, 2:58 PM

We're happy to see #ApachePinot now listed on the #TWTechRadar as one of the top platforms to assess. This is a big accomplishment for the entire Pinot community. Thanks to everyone that helped us get there! https://twitter.com/ApachePinot/status/1381983417605238789?s=20

🎉 5

👏 7

Karin Wolok

04/13/2021, 5:45 PM

📣 Just a reminder! 🎉 If anyone is interested in presenting at the Apache Pinot event series, please submit today! Presentations will be scheduled in May, June, July. Topics will be a variety on use cases, how-to's, your experiences working with Pinot, "getting started with X in Pinot", features and connectors. There's really no limit with types of topics and it can be a work in progress! Even if your use case isn't fully built out, many people might be interested to see what are you working on, what made you think of Pinot, what were you doing before, what led you here, what works for you and what doesn't, how you compared your options, comparisons of Pinot and other solutions, etc. Feel free to reach out to me if you have questions! https://bit.ly/pinotspeaker

Karin Wolok

04/13/2021, 8:50 PM

Welcome new 🍷 Pinot members! 👋 Tell us about yourselves! How'd you find the community? What are you working on? @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

Gabriel Lucano

04/14/2021, 5:23 PM

Hey guys, What is the correct way to add a table/schema from kafka via UI?

Ming Liang

04/15/2021, 10:54 PM

Copy code

Failed to execute goal on project pinot-confluent-avro: Could not resolve dependencies for project org.apache.pinot:pinot-confluent-avro:jar:0.8.0-SNAPSHOT: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.3.1: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.3.1: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:5.3.1 from/to maven-default-http-blocker (<http://0.0.0.0/>): Blocked mirror for repositories: [confluent (<http://packages.confluent.io/maven/>, default, releases+snapshots)] -> [Help 1]
[ERROR]

Erjan G.

04/26/2021, 12:55 PM

hi guys, what is the difference btw Apache Kudu and Pinot? use cases?

Karin Wolok

04/27/2021, 3:23 PM

🍷 Help us welcome our newest community members! pinot Hello! 👋 🎉 Tell us who you are and what brought you here! 😃 @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

🙌 4

🍷 4

👋 2

Karin Wolok

04/27/2021, 7:00 PM

If there's anyone in the community that's interested in doing some "Getting Started with X in Apache Pinot" type content for others in the community, please let me know! 🙂

🍷 1

kauts shukla

04/28/2021, 6:24 AM

Hello, I have 2 nodes each as controller, broker & server under default tenant but under tenants [default tenant] is show only one broker, one server why ?

kauts shukla

05/02/2021, 11:24 AM

This is the schema

Copy code

{
  "schemaName": "click_stream",
  "dimensionFieldSpecs": [
    {
      "name": "id",
      "dataType": "STRING"
    },
    {
      "name": "ipaddress",
      "dataType": "STRING"
    },
    {
      "name": "city",
      "dataType": "STRING"
    },
    {
      "name": "region",
      "dataType": "STRING"
    },
    {
      "name": "country",
      "dataType": "STRING"
    },
    {
      "name": "continent",
      "dataType": "STRING"
    },
    {
      "name": "postalcode",
      "dataType": "STRING"
    },
    {
      "name": "latitude",
      "dataType": "DOUBLE"
    },
    {
      "name": "longitude",
      "dataType": "DOUBLE"
    },
    {
      "name": "isp",
      "dataType": "STRING"
    },
    {
      "name": "useragent",
      "dataType": "STRING"
    },
    {
      "name": "browsername",
      "dataType": "STRING"
    },
    {
      "name": "browserversion",
      "dataType": "STRING"
    },
    {
      "name": "osname",
      "dataType": "STRING"
    },
    {
      "name": "osversion",
      "dataType": "STRING"
    },
    {
      "name": "devicename",
      "dataType": "STRING"
    },
    {
      "name": "deviceversion",
      "dataType": "STRING"
    },
    {
      "name": "clientid",
      "dataType": "STRING"
    },
    {
      "name": "loggedin",
      "dataType": "STRING"
    },
    {
      "name": "grade",
      "dataType": "STRING"
    },
    {
      "name": "target",
      "dataType": "STRING"
    },
    {
      "name": "board",
      "dataType": "STRING"
    },
    {
      "name": "devicetype",
      "dataType": "STRING"
    },
    {
      "name": "referrer",
      "dataType": "STRING"
    },
    {
      "name": "utm_source",
      "dataType": "STRING"
    },
    {
      "name": "utm_medium",
      "dataType": "STRING"
    },
    {
      "name": "utm_campaign",
      "dataType": "STRING"
    },
    {
      "name": "utm_term",
      "dataType": "STRING"
    },
    {
      "name": "utm_content",
      "dataType": "STRING"
    },
    {
      "name": "channel",
      "dataType": "STRING"
    },
    {
      "name": "sessionid",
      "dataType": "STRING"
    },
    {
      "name": "domain",
      "dataType": "STRING"
    },
    {
      "name": "fullurl",
      "dataType": "STRING"
    },
    {
      "name": "pagepath",
      "dataType": "STRING"
    },
    {
      "name": "pagetitle",
      "dataType": "STRING"
    },
    {
      "name": "screenname",
      "dataType": "STRING"
    },
    {
      "name": "type",
      "dataType": "STRING"
    },
    {
      "name": "appname",
      "dataType": "STRING"
    },
    {
      "name": "apptrackingid",
      "dataType": "STRING"
    },
    {
      "name": "screenresolution",
      "dataType": "STRING"
    },
    {
      "name": "viewportsize",
      "dataType": "STRING"
    },
    {
      "name": "eventcategory",
      "dataType": "STRING"
    },
    {
      "name": "eventaction",
      "dataType": "STRING"
    },
    {
      "name": "eventlabel",
      "dataType": "STRING"
    },
    {
      "name": "k1",
      "dataType": "STRING"
    },
    {
      "name": "v1",
      "dataType": "STRING"
    },
    {
      "name": "k2",
      "dataType": "STRING"
    },
    {
      "name": "v2",
      "dataType": "STRING"
    },
    {
      "name": "k3",
      "dataType": "STRING"
    },
    {
      "name": "v3",
      "dataType": "STRING"
    },
    {
      "name": "k4",
      "dataType": "STRING"
    },
    {
      "name": "v4",
      "dataType": "STRING"
    },
    {
      "name": "k5",
      "dataType": "STRING"
    },
    {
      "name": "v5",
      "dataType": "STRING"
    },
    {
      "name": "k6",
      "dataType": "STRING"
    },
    {
      "name": "v6",
      "dataType": "STRING"
    },
    {
      "name": "k7",
      "dataType": "STRING"
    },
    {
      "name": "v7",
      "dataType": "STRING"
    },
    {
      "name": "k8",
      "dataType": "STRING"
    },
    {
      "name": "v8",
      "dataType": "STRING"
    },
    {
      "name": "k9",
      "dataType": "STRING"
    },
    {
      "name": "v9",
      "dataType": "STRING"
    },
    {
      "name": "k10",
      "dataType": "STRING"
    },
    {
      "name": "v10",
      "dataType": "STRING"
    },
    {
      "name": "useremail",
      "dataType": "STRING"
    },
    {
      "name": "firstpagetitle",
      "dataType": "STRING"
    },
    {
      "name": "first_fullurl",
      "dataType": "STRING"
    },
    {
      "name": "deviceuniqueid",
      "dataType": "STRING"
    },
    {
      "name": "deviceid",
      "dataType": "STRING"
    },
    {
      "name": "devicebrand",
      "dataType": "STRING"
    },
    {
      "name": "devicemanufacturer",
      "dataType": "STRING"
    },
    {
      "name": "devicemodel",
      "dataType": "STRING"
    },
    {
      "name": "serialnumber",
      "dataType": "STRING"
    },
    {
      "name": "buildnumber",
      "dataType": "STRING"
    },
    {
      "name": "carrier",
      "dataType": "STRING"
    },
    {
      "name": "batterylevel_new",
      "dataType": "STRING"
    },
    {
      "name": "firstinstalltime",
      "dataType": "INT"
    },
    {
      "name": "isemulator",
      "dataType": "STRING"
    },
    {
      "name": "istablet",
      "dataType": "STRING"
    },
    {
      "name": "islandscape",
      "dataType": "STRING"
    },
    {
      "name": "batterylevel",
      "dataType": "INT"
    },
    {
      "name": "firstinstalltimeist",
      "dataType": "INT"
    },
    {
      "name": "firstopened_datetime",
      "dataType": "LONG"
    },
    {
      "name": "daysfrom_firstopened",
      "dataType": "INT"
    },
    {
      "name": "doubtactionablecount",
      "dataType": "INT"
    },
    {
      "name": "gapp_userstatus",
      "dataType": "STRING"
    },
    {
      "name": "gaid",
      "dataType": "STRING"
    },
    {
      "name": "last_attributed_touch_type",
      "dataType": "STRING"
    },
    {
      "name": "last_attributed_touch_timestamp",
      "dataType": "STRING"
    },
    {
      "name": "last_attributed_touch_timestamp_iso",
      "dataType": "LONG"
    },
    {
      "name": "last_attributed_touch_data_tilde_campaign_id",
      "dataType": "STRING"
    },
    {
      "name": "last_attributed_touch_data_tilde_branch_ad_format",
      "dataType": "STRING"
    },
    {
      "name": "last_attributed_touch_data_tilde_technology_partner",
      "dataType": "STRING"
    },
    {
      "name": "last_attributed_touch_data_tilde_banner_dimensions",
      "dataType": "STRING"
    },
    {
      "name": "firsttransactiondate",
      "dataType": "LONG"
    },
    {
      "name": "gpslatitude",
      "dataType": "DOUBLE"
    },
    {
      "name": "gpslongitude",
      "dataType": "DOUBLE"
    },
    {
      "name": "gpscity",
      "dataType": "STRING"
    },
    {
      "name": "source",
      "dataType": "STRING"
    },
    {
      "name": "version",
      "dataType": "STRING"
    },
    {
      "name": "userid",
      "dataType": "STRING"
    },
    {
      "name": "calc_userid",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "timestampist",
      "dataType": "LONG"
    }
  ],
  "timeFieldSpec": {
    "incomingGranularitySpec": {
      "name": "created_at",
      "dataType": "LONG",
      "timeType": "MILLISECONDS"
    }
  },
  "primaryKeyColumns": [
    "userid",
    "eventcategory",
    "eventlabel"
  ]
}

kauts shukla

05/02/2021, 11:28 AM

It’s not a upsert table. the table config is which I’m using….

Copy code

{
  "REALTIME": {
    "tableName": "click_stream_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "timeColumnName": "created_at",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "90",
      "segmentPushType": "APPEND",
      "timeType": "DAYS",
      "replication": "1",
      "replicasPerPartition": "1",
      "schemaName": "click_stream"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "rangeIndexColumns": [
        "created_at",
        "timestampist"
      ],
      "autoGeneratedInvertedIndex": false,
      "sortedColumn": [
        "userid",
        "eventcategory",
        "eventlabel"
      ],
      "loadMode": "MMAP",
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "simple",
        "stream.kafka.topic.name": "click_stream",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.hlc.zk.connect.string": "zk1:2181/kafka",
        "stream.kafka.zk.broker.url": "zk1:2181/kafka",
        "stream.kafka.broker.list": "kfk1:9092",
        "realtime.segment.flush.threshold.time": "3600000",
        "realtime.segment.flush.threshold.size": "50000",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
      },
      "aggregateMetrics": true,
      "enableDefaultStarTree": true,
      "enableDynamicStarTreeCreation": true,
      "nullHandlingEnabled": false,
      "invertedIndexColumns": [
        "userid",
        "eventcategory",
        "eventlabel"
      ],
      "createInvertedIndexDuringSegmentGeneration": true
    },
    "metadata": {
      "customConfigs": {}
    },
    "routing": {
      "instanceSelectorType": "replicaGroup"
    },
    "isDimTable": false
  }
}

Syed Akram

05/10/2021, 7:50 AM

check this for ref

Pedro Silva

05/11/2021, 11:13 AM

What are the consequences of defining a dimension as a String of length Integer.MaxSize and creating a json index based on it? What hardware considerations need to be made if any? What is the performance impact?

Karin Wolok

05/11/2021, 4:00 PM

👋 Let's give a big warm welcome to all the new Pinot community members! 🍷 👉 We'd love to learn a little bit about who you are and what brought you here! Please take a moment and introduce yourself to the community! ❤️ @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

vishal sharma

05/11/2021, 4:06 PM

Hi, thanks for having us here. I am a software developer from India, just started in the tech industry last year, graduating in 2020. My field of interest is distributed systems and i work majorly on go.

❤️ 1

👋 3

Ambika

05/12/2021, 8:19 AM

Another question -- • How does presto push down aggregations when we do joins ? If it doesn't push down and it fetches the row to do the agg, then it will be slow. My use case is a typical star schema (a fact table and multiple dimension tables). The slice/dice will generally occur on top of these dimensions for which i will need join capabilities. (sorry getting questions as I am doing some tests and thinking about my usecase)

Mayank

05/17/2021, 2:41 PM

Hmm, can you file an issue? Also check if restarting helps

05/17/2021, 2:44 PM

Restarted everything , and tried but still no luck

05/17/2021, 2:57 PM

Is there any other way to update this server instance so it can reach port 8098 instead of 8008?

Badri Tripathy

05/17/2021, 4:55 PM

Hello everyone... We are new to Pinot and exploring the OLAP possibilities. We want to increase the performance of some of our dashboard queries running on Presto involving typical joins ,aggregations and filters. We already have dimensions and Facts on S3/Presto and use spark to populate those. Is it worth to pre compute in 1 more layer and push it to Pinot for slice and dice only with fat table structure or use Presto to join the Pinot offline tables. How you guys have solved the problem.Any pointers is appreciated.

05/18/2021, 6:40 AM

Is there any way to handle null values in Pinot. I.e. I have one column in Kafka in some messages it contains some value and sometime it's coming null. I have added this column in transformation function so in this case it only showing those row in which we have value for this column. And I I don't add this column in transformation function in that case it's showing all the rows but for this column taking null in all the rows.

Lochanie

05/18/2021, 9:13 AM

Hi All, I am a complete beginner to apache pinot. Today I have installed apache pinot in my windows 10 machine using below link. https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally. But when I type http://localhost:9000 , I cannot see the pinot data explore. It gives the error message this site cant be reached. Please help me to resolve this issue. my pinot version is 0.6.0 and JDK version is 8. I ran the bach quick-start command only. Thank you!

Karin Wolok

05/20/2021, 1:42 PM

🍷 New to Apache Pinot and want to understand the basics? Join us for Intro to Apache Pinot meetup today! 10am PDT | 1pm EDT 🙂 https://www.meetup.com/apache-pinot/events/277817386/

🙌 1

Aditya S

05/26/2021, 1:58 AM

Hi Everyone. I have created a Udemy course on Apache Pinot. Feel free to check the course out https://www.udemy.com/course/apache-pinot-a-hands-on-course/?referralCode=67F5295AA86AF59C00E4

🎉 8

🍷 4

Kenny Bastani

05/26/2021, 11:41 AM

set the channel topic: Welcome to the community! Please show your support by starring our project repository on GitHub https://github.com/apache/incubator-pinot