https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • t

    Ting Chen

    08/14/2019, 6:09 PM
    https://github.com/apache/incubator-pinot/issues/4525
  • t

    Ting Chen

    08/14/2019, 6:11 PM
    We filed an issue here for more details. One observation is that Pinot server id is now formatted as Server_<host>_<port> which is coupled with http hostname.
  • x

    Xiang Fu

    08/14/2019, 6:16 PM
    yes, I make a change to make it be able to take dns hostname not only ip:
    <https://github.com/apache/incubator-pinot/pull/4517>
  • x

    Xiang Fu

    08/14/2019, 6:16 PM
    kubernetes could provide dns name like pinot-server-0/pinot-server-1/pinot-server-2
  • x

    Xiang Fu

    08/14/2019, 6:16 PM
    per container
  • x

    Xiang Fu

    08/14/2019, 6:17 PM
    for uber case, maybe we could open env_variable to override hostname
  • x

    Xiang Fu

    08/14/2019, 6:18 PM
    so you can keep consistent instance_id after deployment
  • m

    Mayank

    08/14/2019, 6:18 PM
    We are starting an effort to run Pinot on Azure at LinkedIn. However, for the initial phase it will be via LinkedIn deployment stack.
  • x

    Xiang Fu

    08/14/2019, 6:18 PM
    even without stickness
  • m

    Mayank

    08/14/2019, 6:19 PM
    I anticipate we might face similar issues, and would like to ensure that we propose a generic solution that works for all causes, as opposed to being tied to a specific deployment system.
  • m

    Mayank

    08/14/2019, 6:19 PM
    Thoughts welcome
  • x

    Xiang Fu

    08/14/2019, 6:20 PM
    correct
  • t

    Ting Chen

    08/14/2019, 6:20 PM
    this looks promising @User, definitely a logical service name like pinot-server-2 will help.
  • x

    Xiang Fu

    08/14/2019, 6:21 PM
    I feel the most generic way is that we should let pinot accept an external passing instance_id from outside other than config file
  • x

    Xiang Fu

    08/14/2019, 6:23 PM
    we need a logical id to dns mapping to support this.
  • m

    Mayank

    08/14/2019, 6:24 PM
    Agree with the logical id. Not sure how it would work, for example, if a VM dies and is replaced by another with a different ip address, how to ensure it gets the same logical id?
  • x

    Xiang Fu

    08/14/2019, 6:25 PM
    the logical id should be associated and provided by any deployment framework.
  • m

    Mayank

    08/14/2019, 6:25 PM
    Ah, I see
  • x

    Xiang Fu

    08/14/2019, 6:25 PM
    e.g, uber deployment will say i have a pinot-server deployment with 100 servers
  • x

    Xiang Fu

    08/14/2019, 6:26 PM
    or 100 instances
  • x

    Xiang Fu

    08/14/2019, 6:26 PM
    server/port could change
  • x

    Xiang Fu

    08/14/2019, 6:26 PM
    but you will always have pinot-server-0 to pinot-server-99
  • t

    Ting Chen

    08/14/2019, 6:27 PM
    in Uber right now, we can control the server instance id through config. @User
  • f

    faiyaz

    08/14/2019, 6:30 PM
    i tried out using a custom name like Node01- Node 02 etc. however it is not possible to update DNS on-demand or create cname etc for Node01 / Node 02 etc. so broker is not able to route query to correct hosts.
  • x

    Xiang Fu

    08/14/2019, 6:31 PM
    one way is to get the dns mapping from helix, I think the mapping is there
  • t

    Ting Chen

    08/14/2019, 6:33 PM
    right, it is in the instance config of helix zk now but the mapping is not used for building routing tables -- only the server instance ids are used now.
  • t

    Ting Chen

    08/14/2019, 6:43 PM
    @User your PR works for kubernetes because K8s provides DNS. Uber does not have DNS for now AFAIK. So we probably still need Pinot to perform logical_id->hostname lookup.
  • x

    Xiang Fu

    08/14/2019, 6:43 PM
    yes
  • x

    Xiang Fu

    08/14/2019, 6:43 PM
    I think that change will happen in broker side
  • x

    Xiang Fu

    08/14/2019, 6:44 PM
    need to abstract dns mapping part
1...808182...160Latest