https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • k
    everytime aggregate is called, pinot passes 5k to 10k values at a time
  • d
    ok and PInot calls merge() it means that all the blocks have been processed and they could be merged.. right? but wait..
  • d
    is merge() called globally or only with the blocks of the segment?
  • d
    i mean, e.g. a segment has 10 blocks, Pinot calls 10 times aggregate() and 10 times merge() ?
  • d
    if yes...this is a problem, because i need something globally... extractFinalResult maybe?
  • d
    i mean globally because a max could be in a segment and the min in another segment as i told you i need to keep the bigger max-min difference of the ranges
  • k
    no
  • k
    think 2 servers, 10 segments in each server, 100 blocks in each segment
  • k
    aggregate is called 100 times for each segment
  • k
    merge is called 10 times (once per segment, technically 9 bcos there is nothing to merge the first time)
  • k
    merge is called again once in broker, to merge the results from two servers
  • d
    ok ok perfect...so i can get ALL the ranges in merge() at the final step
  • d
    in that wait i can see if a range has been splitted inside two servers
  • d
    maybe a parte of the range in serverA and the other in the serverB
  • k
    yeah, one inefficient solution is to hold on to all data points and do the logic in extractFinalResult
  • k
    you will have everything in the end
  • d
    why inefficient? i can move the logic from merge() to extractFinalResult()
  • d
    yeah the problem is having that big list of ranges in extractFinalResult()..i should find a way to reduce it...
  • k
    right
  • k
    but that would be my first implementation
  • k
    get it working for one ticker
  • k
    or one key
  • k
    select dropDown(time, price) from T where ticker='X'
  • d
    and then try the group by
  • d
    ok
  • k
    yeah
  • d
    Xiang was able to help me solve my Presto problem. Thanks Xiang!
    🎉 4
  • d
    Copy code
    % kubectl logs -n ${NAMESPACE} pod/pinot-broker-0  --previous
2020/05/04 04:46:55.793 INFO [HelixBrokerStarter] [main] Starting Pinot broker
2020/05/04 04:46:55.803 INFO [HelixBrokerStarter] [main] Connecting spectator Helix manager
2020/05/04 04:46:56.474 INFO [HelixBrokerStarter] [main] Setting up broker server builder
May 04, 2020 4:47:02 AM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8099]
May 04, 2020 4:47:03 AM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
2020/05/04 04:47:28.713 INFO [HelixBrokerStarter] [main] Initializing cluster change mediator
2020/05/04 04:47:28.894 INFO [HelixBrokerStarter] [main] Connecting participant Helix manager
2020/05/04 04:47:28.964 WARN [ParticipantHealthReportTask] [main] ParticipantHealthReportTimerTask already stopped
2020/05/04 04:47:29.601 WARN [CallbackHandler] [main] Callback handler received event in wrong order. Listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6a8b8de, path: /pinot-quickstart/INSTANCES/Broker_pinot-broker-0.pinot-broker-headless.events-local.svc.cluster.local_8099/MESSAGES, expected types: [CALLBACK, FINALIZE] but was INIT
2020/05/04 04:47:30.034 INFO [HelixBrokerStarter] [main] Registering service status handler
2020/05/04 04:47:30.308 WARN [ConfigAccessor] [ZkClient-EventThread-25-pinot-zookeeper:2181] No config found at /pinot-quickstart/CONFIGS/RESOURCE/brokerResource
2020/05/04 04:47:30.335 INFO [HelixBrokerStarter] [main] Finish starting Pinot broker
2020/05/04 04:47:31.023 ERROR [BaseRoutingTableBuilder] [HelixTaskExecutor-message_handle_thread] Found no server hosting segment events__0__0__20200502T0226Z for table events_REALTIME
2020/05/04 04:51:01.182 INFO [HelixBrokerStarter] [Thread-1] Shutting down Pinot broker
2020/05/04 04:51:01.182 INFO [HelixBrokerStarter] [Thread-1] Shutting down time boundary info refresh message handler
2020/05/04 04:51:01.182 INFO [HelixBrokerStarter] [Thread-1] Disconnecting participant Helix manager
2020/05/04 04:51:02.527 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:02.579 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:02.584 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:02.600 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:02.600 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:02.600 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:03.739 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:03.739 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:03.739 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:03.741 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:03.741 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:03.741 WARN [StateModel] [Thread-1] Default reset method invoked. Either because the process longer own this resource or session timedout
2020/05/04 04:51:07.770 INFO [HelixBrokerStarter] [Thread-1] Stopping cluster change mediator
2020/05/04 04:51:07.800 INFO [HelixBrokerStarter] [Thread-1] Stopping broker server builder
2020/05/04 04:51:17.978 ERROR [DataTableHandler] [nioEventLoopGroup-2-1] Channel for server: pinot-server-0_O is now inactive, marking server down
2020/05/04 04:51:21.199 INFO [HelixBrokerStarter] [Thread-1] Disconnecting spectator Helix manager
2020/05/04 04:51:22.507 INFO [HelixBrokerStarter] [Thread-1] Finish shutting down Pinot broker
  • d
    I'm running Pinot locally using Kubernetes and the pinot-broker-0 keeps restarting. The logs don't have many details.
  • d
    Hello
1...949596...166Latest