https://pinot.apache.org/ logo
Join Slack
Powered by
# metadata-push-api
  • m
    And use SSD
  • s
    exactly
  • s
    SSD for sure if we are not able to get rid of untarring business
  • m
    Will that solve our problem
  • x
    I somehow feel the bottleneck is the controller download, if we can bypass this path, then controller overhead will be reduced a lot
  • m
    Yes controller bypassing will help a lot
  • m
    Unfortunately at lnkd we don’t have a reliable deepstore
  • x
    we can add an option to convert metadata.properties and creation.meta file to json and let controller to handle that as well
  • x
    basically there is no need for controller to download the data part
  • x
    even current just metadata tar file helps a lot on the controller side
  • m
    But we don’t have a deepstore that is reliable and can be used in prod
  • m
    So we have no way to bypass controller
  • x
    ic
  • x
    not even hdfs?
  • m
    Availability is not good enough for production
  • x
    pinot segments are anyway generated there and stored right?
  • x
    ic
  • x
    even offline push is not good ?
  • m
    Offline push is good
  • x
    for realtime side
  • x
    Uber has implemented the p2p download
  • x
    maybe you wanna try out
  • m
    So we are trying out ideas for deepstore
  • x
    so servers can download from each other
  • x
    got it
  • m
    But in case none of those work we want to still optimize with controller in the path
  • x
    maybe 2 hdfs 🙂
  • x
    hdfs1:// and hdfs2://
  • x
    I remember at Uber they used to use two hdfs to satisfy the availability requirements 😛
  • m
    😃