https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • j

    James Shao

    04/01/2020, 9:31 PM
    hi, I am just wondering if anyone seen pinot broker has latency spike more than pre-defined timeout period when the request rate/latency is high
  • j

    James Shao

    04/01/2020, 9:32 PM
    for example, if broker got hundreds of requests per second with each request take seconds to finish, will pinot broker web server runs out of request handler and cause all incoming request to stuck in the some kind of pending state?
  • k

    Kishore G

    04/01/2020, 9:33 PM
    after timeout the connection handles must be released and available to use for new queries
  • j

    James Shao

    04/01/2020, 9:37 PM
    yes, but I saw that in broker we have connection pool with default something like 30?
  • j

    James Shao

    04/01/2020, 9:38 PM
    if the broker -> server requests take long time to finish and we have high qps to the broker, will the broker runs out all request handler and cause all incoming request to this broker to get into the “waiting state”?
  • k

    Kishore G

    04/01/2020, 9:39 PM
    yes, @User added the support for multiplexing over the same connection, not sure if that takes care of this scenario
  • j

    James Shao

    04/01/2020, 9:41 PM
    got it. do you know if we have existing metrics exposed for monitoring such scenario? for example java jetty server has some metrics like available connection count metrics to let us get to know if we still have enough connection handler on the java web server
  • m

    Mayank

    04/01/2020, 9:41 PM
    With the new connection pool, we have the reverse effect where broker bombards servers in a non blocking way
  • m

    Mayank

    04/01/2020, 9:42 PM
    IIRC, the current code will drop queries that have already passed the timeout
  • m

    Mayank

    04/01/2020, 9:42 PM
    The default value of timeout may be too big (30 sec??), and tuning that should mitigate that issue
  • m

    Mayank

    04/01/2020, 9:43 PM
    There is a metric to monitor the wait time in broker queue
  • j

    James Shao

    04/01/2020, 9:43 PM
    got it, thanks
  • j

    James Shao

    04/01/2020, 9:43 PM
    will look into that for better monitoring
  • j

    Jackie

    04/01/2020, 11:14 PM
    I think we had this metric to track the dropped request:
    REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR
  • g

    Gaurav Dawra

    04/02/2020, 4:45 AM
    Hi, we were wondering if anyone has integrated Grafana and whether Grafana can be used as a visualization tool for data/metrics that we have plans to store in Pinot.
    👍 1
  • k

    Kishore G

    04/02/2020, 4:51 AM
    we haven't tried that, there is a superset adapter
  • m

    Mayank

    04/02/2020, 4:52 AM
    @User
  • a

    Ananth Packkildurai

    04/02/2020, 4:53 AM
    I’m talking to @User on DM, We actually integrated Pinot metrics, not the grafana-pinot integration. It would be a nice integration, especially now Pinot support Lucene integration
    👍 1
  • d

    Dan Hill

    04/02/2020, 9:37 PM
    I'm a bit confused how about how to query a table and filter to specific time ranges. I have a timeFieldSpec that specifies a timestamp column, named
    timestamp
    and a segmentsConfig that specifies a timestamp column. When I try to query the timestamp column in PQL, I get an error
    Encountered "timestamp >"
    ...
    Was expecting one of
    . Do I need to refer to a different name? Do I need to add dimension fields for the parts of date I want to query? Do I have to use a function to convert timestamp to a timestamp I can filter between?
  • k

    Kishore G

    04/02/2020, 9:39 PM
    timestamp is a reserved keyword
  • s

    Subbu Subramaniam

    04/02/2020, 9:39 PM
    I think timestamp is a reserved keyword in SQL, so you will need to quote the column name in the query
  • d

    Dan Hill

    04/02/2020, 9:40 PM
    Ah, that's it. Thanks!
  • d

    Dan Hill

    04/02/2020, 10:38 PM
    If I filter for timestamps outside of the data range, my request gives a
    ProcessingException(errorCode: 450,
    . Is that intended? I figured it'd just return empty rows.
  • m

    Mayank

    04/02/2020, 10:39 PM
    No, it should return your empty rows if no rows are selected
  • m

    Mayank

    04/02/2020, 10:39 PM
    The error implies something else went wrong
  • d

    Dan Hill

    04/02/2020, 11:33 PM
    Hmm, all I changed was a timestamp filter to an earlier date.
  • k

    Kishore G

    04/03/2020, 1:03 AM
    Do you have queries?
  • d

    Dan Hill

    04/03/2020, 4:56 PM
    Hmm, weird, I can't reproduce it now. If/when I hit it again, I'll record them.
  • d

    Dan Hill

    04/03/2020, 4:58 PM
    Unrelated, has anyone hooked up a Pinot Connector to to AWS Athena (managed Presto)? No worries if not.
  • k

    Kishore G

    04/03/2020, 4:59 PM
    no, is that even possible?
1...121122123...160Latest