https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • k
  • s
  • s
    hi ，When to execute the expired data, 6*3600?
  • s
    How do you execute the pinot sever down line? How does the data above migrate?
  • x
    controller has a retention manager which will kick off retention job
  • x
    default is 6 hours
  • x
    for server migration you can use segment rebalance tool
  • s
    In some tables（realtime table and offline table）, the data segment is OFFLINE, so how can it be transferred to the ONLINE state
  • s
    image.png
  • m
    Are you suspecting the segment is offline due to retention, or is this an issue unrelated to your previous question?
  • m
    If you have access to the server log, it would likely indicate why the segment is offline (typically due to some error).
  • s
    It’s probably because of retention, but how do you delete those OFFLINE segments
  • s
    I find that rebalance depends on the /tmp/PinotController directory under the controller. What if the amount of data is too large for the /tmp/PinotController directory disk? Because /tmp/PinotController takes over all the data segments of the cluster
  • m
    You shouldnt need to delete them manually
  • m
    Do you mean rebalance or retention?
  • m
    Will check the code and get back
  • m
    @User Are you starting the controller with default configs? You can (and should) overwrite the 
    controller.data.dir
    config to a location that has enough storage, and can also be shared across multiple instances of the controller (in production). For example, in production you can use a deep-storage such as S3/HDFS/ADLS, or NFS that is mounted on the controller instances.
  • e
    Re: new star tree index config - We wanted to make sure we have the new config. This is our current star tree config (example):
    Copy code
    "starTreeIndexSpec": {
      "dimensionsSplitOrder": [
        "column1",
        "column2",
        "column3"
      ],
      "skipStarNodeCreationForDimensions": [],
      "functionColumnPairs": [
        "COUNT__columnX",
        "SUM__columnX",
        "MAX__columnX",
        "AVG__columnX",
        "COUNT__columnY",
        "SUM__columnY",
        "MAX__columnY",
        "AVG__itemQuantity",
        "COUNT__tax",
        "SUM__tax",
        "MAX__tax",
        "AVG__tax"
      ]
    }
  • e
    @User ^^^ is that the new config?
  • j
    @User This is a mix of both old and new configs... The new configs are under the key 
    starTreeIndexConfigs
    and takes a list of configs (one for each tree)
  • j
    Copy code
    "starTreeIndexConfigs": [
  {
    "dimensionsSplitOrder": [
      "column1",
      "column2",
      "column3"
    ],
    "functionColumnPairs": [
      "COUNT__columnX",
      "SUM__columnX",
      "MAX__columnX",
      "AVG__columnX",
      "COUNT__columnY",
      "SUM__columnY",
      "MAX__columnY",
      "AVG__itemQuantity",
      "COUNT__tax",
      "SUM__tax",
      "MAX__tax",
      "AVG__tax"
    ]
  }
]
  • e
    Perfect thanks! Also what is the
    Copy code
    maxLeafRecords
    config?
  • j
    That is the upper bound of the records to process for each leaf node (default 10000)
  • j
    The lower the threshold, the less records to process (higher performance), but that will increase the size of the tree.
  • j
    You may read more about how star-tree works here: https://pinot.readthedocs.io/en/latest/star-tree/star-tree.html
    👍 1
  • s
    in production you can use a deep-storage HDFS, Does this mount the HDFS specific document @User
  • s
    or Does this mount the NFS specific document
  • m
    Not sure if I follow the question @User
  • m
    The controller data.dir can only be one of these storages.
  • m
    If you specify data.dir on a hdfs:// path, then controller will use that to store segments and servers will download from there.
1...115116117...160Latest