https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • x

    Xiang Fu

    08/14/2019, 6:44 PM
    currently I think we just parse the instance id
  • t

    Ting Chen

    08/14/2019, 6:46 PM
    that is right.
  • t

    Ting Chen

    08/14/2019, 6:49 PM
    as Mayank said, we need a general solution for this. We will file a proposal.
  • k

    Kishore G

    08/14/2019, 7:14 PM
    I thought we are already using the host name and port from Helix on the broker side
  • k

    Kishore G

    08/14/2019, 7:15 PM
    are we deriving host name and port from instanceId?
  • k

    Kishore G

    08/14/2019, 7:15 PM
    that should be a simple change
  • t

    Ting Chen

    08/14/2019, 7:18 PM
    yes we derive the name port from instanceId string.
  • t

    Ting Chen

    08/14/2019, 7:19 PM
    We can do it Helix look up to get the right hostname and port -- this will be enough for us.
  • x

    Xiang Fu

    08/14/2019, 7:21 PM
    Copy code
    /**
       * Given an external view and a list of instance configs, computes the mapping of segment to servers. The mapping
       * will be cached if we use dynamic routing. By default, this will check ONLINE segments and active servers.
       *
       * @param externalView an external view
       * @param instanceConfigs a list of instance config
       * @return a mapping of segment to servers
       */
      protected Map<String, List<String>> computeSegmentToServersMapFromExternalView(ExternalView externalView,
          List<InstanceConfig> instanceConfigs) {
        Map<String, List<String>> segmentToServersMap = new HashMap<>();
        RoutingTableInstancePruner instancePruner = new RoutingTableInstancePruner(instanceConfigs);
        for (String segmentName : externalView.getPartitionSet()) {
          // List of servers that are active and are serving the segment
          List<String> servers = new ArrayList<>();
          for (Map.Entry<String, String> entry : externalView.getStateMap(segmentName).entrySet()) {
            String serverName = entry.getKey();
            if (entry.getValue().equals(<http://CommonConstants.Helix.StateModel.SegmentOnlineOfflineStateModel.ONLINE|CommonConstants.Helix.StateModel.SegmentOnlineOfflineStateModel.ONLINE>)
                && !instancePruner.isInactive(serverName)) {
              servers.add(serverName);
            }
          }
          if (!servers.isEmpty()) {
            segmentToServersMap.put(segmentName, servers);
          } else {
            handleNoServingHost(segmentName);
          }
        }
        return segmentToServersMap;
      }
  • x

    Xiang Fu

    08/14/2019, 7:21 PM
    server is from instance id
  • s

    Subbu Subramaniam

    08/14/2019, 8:01 PM
    the servers need to look up the controllers for segment download, and the controllers need to advertise a host/port outside for segment push, and general API as well. Brute force solution can be to make the instance name configurable (like in broker) and provide a mapping (in the cloud, deployment framework, whatever). Not sure why @User wanted the instance name to be outside of config.
  • x

    Xiang Fu

    08/14/2019, 8:07 PM
    what i mean is that we should have a way other than just file based config.
  • t

    Ting Chen

    08/14/2019, 9:14 PM
    thanks @User. Yes, that is the code which uses External view to construct routing map and external view only has serve names. In general, we propose (1) Pinot broker should use the server instance id to look up the server address from Helix/Zk instead of parsing from server id (2) Pinot server should decouple the configuration of server Id with server host/port. For more details: please take a look at my comments in https://github.com/apache/incubator-pinot/issues/4525. Comments welcome.
    👍 2
  • c

    Chinmay Soman

    08/15/2019, 6:45 PM
    +1 to not parsing instance name for discovering servers. That seems hacky.
  • d

    Dheeraj Dwivedi

    08/21/2019, 7:21 AM
    Need help with pinot vs druid benchmark scripts. Where will I be able to find data files like: pinot_non_startree.tar.gz ... not able to find in repo ?
  • k

    Kishore G

    08/21/2019, 7:35 AM
    you can generate the data using tpch tool
  • a

    abk

    08/22/2019, 3:13 PM
    Hey guys, some qns: is Pinot comparable to Druid? Is there any good comparative study that I can refer to?
  • a

    abk

    08/22/2019, 3:14 PM
    We have some must-have features like ingesting parquet files, columnar, scalable and can talk to superset (integration) for visualizations.
  • a

    abk

    08/22/2019, 3:14 PM
    If you could point me to to the right links , will be of great help
  • k

    Kishore G

    08/22/2019, 3:15 PM
    see this medium blog post : https://medium.com/@leventov/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7
    👍 1
  • k

    Kishore G

    08/22/2019, 3:17 PM
    yes, you can ingest parquet files.Superset integration was added recently
  • a

    abk

    08/22/2019, 3:18 PM
    Thanks much.
  • k

    Kishore G

    08/22/2019, 3:19 PM
    https://docs.google.com/presentation/d/1yi2b-otE1eBpJ8MzX_cFfA7oB1ThdMDgHESLx53gjvE/edit#slide=id.g5113f4b6b5_2_383
  • k

    Kishore G

    08/22/2019, 3:19 PM
    you can see some perf numbers
  • a

    abk

    08/22/2019, 3:20 PM
    Need access
  • a

    abk

    08/22/2019, 3:20 PM
    just requested.
  • k

    Kishore G

    08/22/2019, 3:20 PM
    https://docs.google.com/presentation/d/1yi2b-otE1eBpJ8MzX_cFfA7oB1ThdMDgHESLx53gjvE/edit?usp=sharing
  • k

    Kishore G

    08/22/2019, 3:20 PM
    try this
  • a

    abk

    08/22/2019, 3:21 PM
    ditto.
  • a

    abk

    08/22/2019, 3:21 PM
    Screen Shot 2019-08-22 at 10.21.10 AM.png
1...818283...160Latest