https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • m
    Is this for real-time ingestion or offline, or both?
    r
    • 2
    • 7
  • s
    @User I had recently added support for blob like string columns in raw forward index. So physically you can still store them and each column value can go upto millions of characters. But we saw increased heap overhead in the segment generation code path since until it makes into the raw index, the values will be on heap. Secondly, this was used for text search filter so we never selected them and just used for filtering so not sure what the overhead is going to look like on selection path.
    b
    • 2
    • 5
  • e
    Good news! The PR for the Pinot connector is merged (https://github.com/prestosql/presto/pull/2028). It will be available in the upcoming 334 release. It’s compatible with Pinot 0.3.0+ and has a couple of nifty features like query passthrough. I’m working on a blog post to go into more details.
    👍 1
    👏 1
    🎉 7
    k
    k
    • 3
    • 2
  • r
    Question: How can add inverted index to a column programmatically. In current project, Pinot schema is created by uploading file through FileUploadDownloadClient, but I can't figure out how to provide index?
    k
    • 2
    • 5
  • m
    Not that I am aware of, but have wondered the same in the past. Whats your use case for this?
    o
    • 2
    • 1
  • s
    @User and everyone: I'm about to unsubscribe from #CDRCA57FC because the volume of chatter is too high. Can you create a new channel #announce and put announcements there?
    k
    k
    a
    • 4
    • 5
  • s
    @User can you at least change the channel title to "Community-wide announcements _only_"? Personally I'd suggest channel #C01444V4DQC anyway for announcing releases, articles, webinars, events, it's presumably only a matter of time before #CDRCA57FC eventually gets busy with internal (non-troubleshooting) talk. Ideally #C01444V4DQC would have a volume of << 10/month. Thanks
    👍 1
    k
    • 2
    • 2
  • s
    Hi all, we need help implementing a compatibility test suite for pinot. Design document is here: https://docs.google.com/document/d/1yNlvnLKDNUuyRWOKYYF01FWW9weYMGoaLRtU-CueciM/edit?usp=sharing. Code will be merged into master and used on a regular basis, so this can be a nice student project work if they choose to do so.
    m
    c
    +2
    • 5
    • 8
  • b
    Hi all, Pinot broker is logging a warning that it can't find 
    brokerResource
    and then it fails to find servers hosting a segement, though the ideal state and external view clear says the segment is hosted by a server. This setup has only one controller, broker and server running in k8s.
    Copy code
    020/05/21 16:35:00.579 WARN [ParticipantHealthReportTask] [main] ParticipantHealthReportTimerTask already stopped
2020/05/21 16:35:04.476 WARN [ConfigAccessor] [ZkClient-EventThread-27-zookeeper.test.svc.cluster.local:2181/pinot] No config found at /test-views/CONFIGS/RESOURCE/brokerResource
2020/05/21 16:35:04.503 WARN [CallbackHandler] [main] Callback handler received event in wrong order. Listener: org.apache.helix.messaging.handling.HelixTaskExecutor@69d1227f, path: /test-views/INSTANCES/Broker_pinot-broker-0.pinot-broker.test.svc.cluster.local_8099/MESSAGES, expected types: [CALLBACK, FINALIZE] but was INIT
2020/05/21 16:35:04.607 INFO [HelixBrokerStarter] [main] Registering service status handler
2020/05/21 16:35:26.492 WARN [BaseInstanceSelector] [ClusterChangeHandlingThread] Failed to find servers hosting segment: myView__0__0__20200519T1722Z for table: myView_REALTIME (all online instances: [] are disabled)
    who registers the 
    brokerResource
    and when could that be missing?
    k
    • 2
    • 1
  • b
    Hi guys, so, in latest Pinot version, is 
    TEXT_MATCH
    the only way of doing regex like searches on string columns? Are Lucene indexes a must for such functionality? cc @User
    s
    t
    k
    • 4
    • 22
  • k
    @User yes, it was added recently
    👍 1
    e
    • 2
    • 1
  • d
    I'm getting closer to deploying my Pinot to a production stack. I want to run a load test to better understand how to deploy and configure Pinot. Any advice for the load tests? I want to load 50B rows of sample data and (1) measure latencies and (2) iterate on the schema to optimize latencies. I figured I'd deploy a one off AWS cluster for this and iterate on latency tests against it. How many servers do you think I'd need for this? 50B rows. Probably about 10-20 number fields per row.
    k
    • 2
    • 2
  • d
    Are there documents on how to iterate on tables and indices? E.g. adding a new field?
    k
    • 2
    • 2
  • d
    I looked through the Pinot GitBook. Does Pinot have any preference for offline and realtime input format, segment fetchers and processes?
    k
    • 2
    • 2
  • d
    Any preference to camelcase vs snakecase with Pinot tables and columns? The gitbook uses camelcase. Seems kinda weird to see a SQL-like system use camelcase.
    k
    • 2
    • 1
  • c
  • b
    quick question: Where is the 
    pinot-admin.sh
    script that runs as the ENTRYPOINT in the docker image? https://github.com/apache/incubator-pinot/blob/master/docker/images/pinot/Dockerfile#L84
    x
    m
    o
    • 4
    • 14
  • k
    it should return empty with return code 200 (if its not, its a bug)
    o
    • 2
    • 1
  • p
    Hi, not sure if this is the right place, if not please redirect me. I have a question on how pinot uses deep store, IIUC pinot needs the data to be loaded into OfflineServer (or realtime server) to be able to serve queries corresponding to that segment. Does it pull the segment on-demand from deep store if the segment is not already present, i.e. use deep store as another tier of storage? if not are there any plans on including this feature in any future releases?
    k
    k
    • 3
    • 9
  • b
    Is there a Docker image available in Docker hub with the 
    0.4.0
    release candidate? I don't see one
    k
    k
    x
    • 4
    • 10
  • k
    curious, does pinot inherently handle duplicate rows based on some column? • Not as of now.
    k
    p
    • 3
    • 8
  • s
    @User this has to do with an incompatbility between the broker and server for specific types of queries. Assuming you upgrade in the order controller, broker, server, then before upgrading server (but after upgrading broker) you will get errors if these type of queries are issued to the broker. The types of queries are selection with order by i believe, but @User can provide specific examples.
    y
    • 2
    • 5
  • l
    I’m trying to pinot as a single process. I see the following changes are merged https://github.com/apache/incubator-pinot/pull/5266
    s
    • 2
    • 5
  • b
    hey everyone! quick question: Is there any performance difference if I do aggregations on a numeric dimension vs a metric data type?
    m
    • 2
    • 2
  • k
    HI All, Can Pinot write data to kafka which inturn i would like to process and keep it in a datastore...
    m
    s
    k
    • 4
    • 6
  • t
    Copy code
    // Sample avro schema with an aliased field
record Foo {
  string @aliases(["old_bar"]) bar;

  // Other fields
}
    m
    k
    • 3
    • 7
  • s
    Hi all, I was trying realtime ingestion in pinot following the docs. https://docs.pinot.apache.org/basics/getting-started/pushing-your-streaming-data-to-pinot In the query console i am unable to query timestamp field and getting errors. Any idea why is this happening?
    s
    k
    • 3
    • 4
  • n
    I have changed column name to “timstampInEpoch” in the Getting started pages, so that folks don’t hit this error again @User
    s
    • 2
    • 1
  • t
    we found if a STRING column has tab characters in it (i.e., \t) and used as a group-by key. The query result will return two group-by keys for a result. anyone aware of this bug or walk around it?
    n
    j
    m
    • 4
    • 17
  • k
    The article says <1sec for approximately 1B rows. what is the average data size used?
    h
    • 2
    • 3
1...567...160Latest