https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • m
    @User When it was working before, do we know if it was being treated as noDict? I am guessing not.
  • a
    oh, it was fine until last week, since last week the error started showing up.
  • m
    My guess is it was working with dictionary. But something is triggering the size to be zero
  • m
    So the latter still needs to be investigated
  • a
    ☝️ I remember there was an issue with trying to sort in the heap for the no-dict columns.
  • m
    As per the comment we don't support no-dict for String columns for the consuming segments
  • a
    oh ```&& dataType != FieldSpec.DataType.STRING`!!!! 😕
  • m
    Right. So the dictionary was always created I am guessing. Not sure what is causing the size to allocate to be zero.
  • m
    @User Did a new deployment trigger this?
  • a
    it seems so.
  • a
    nope, I can see the code all the way back to 2017.
  • m
    Copy code
    MutableDictionary dictionary = MutableDictionaryFactory
            .getMutableDictionary(dataType, _offHeap, _memoryManager, dictionaryColumnSize,
                Math.min(_statsHistory.getEstimatedCardinality(column), _capacity), allocationContext);
        _dictionaryMap.put(column, dictionary);
  • m
    _statsHistory.getEstimatedCardinality(column)
  • m
    The stats are written into a file on the server
  • m
    May be the stats file is corrupted?
  • m
    @User What do you think?
  • m
    IIRC there's a tool for viewing this stats file
  • a
    to step back, any reason why string data type is not supported for no dict? I thought that is a major differentiator
  • m
    It is not supported only for consuming segments. For consumed segments it is supported.
  • s
    @User i just didn't code it up for string datatype. Contributions welcome 😉
  • a
    ah okay, there are no design issues?
  • s
    is it possible that the column had 0 cardinality in many segments? (i.e. it was never populated in the stream)?
  • a
    hmm, I can think of some edge condition that could happen (maybe a parsing exception). We usually set an empty string for it. Let me double check that as well.
  • a
    yeh, I got some few good points to investigate further. 🙇 @User can you please send me some reference to look on how to add no dict for string when you got a chance.
  • a
    tbh, the low cardinality check seems little rigid since usually we won't have much control over the input stream. It will be great if we can fail safe on these scenarios.
  • s
    the following code is to get estimated carinality:
    Copy code
    public synchronized int getEstimatedCardinality(@Nonnull String columnName) {
    int numEntriesToScan = getNumntriesToScan();
    if (numEntriesToScan == 0) {
      return DEFAULT_EST_CARDINALITY;
    }
    int totalCardinality = 0;
    int numValidValues = 0;
    for (int i = 0; i < numEntriesToScan; i++) {
      SegmentStats segmentStats = getSegmentStatsAt(i);
      ColumnStats columnStats = segmentStats.getColumnStats(columnName);
      if (columnStats != null) {
        totalCardinality += columnStats.getCardinality();
        numValidValues++;
      }
    }
    if (numValidValues > 0) {
      int avgEstimatedCardinality = totalCardinality / numValidValues;
      if (avgEstimatedCardinality > 0) {
        return avgEstimatedCardinality;
      }
    }
    return DEFAULT_EST_CARDINALITY;
  }
  • s
    from what i can see, this always returns DEFAULT_EST_CARDINALITY
  • s
    which is 5000
  • s
  • s
    the code to create dictionary is as:
    Copy code
    MutableDictionary dictionary = MutableDictionaryFactory
            .getMutableDictionary(dataType, _offHeap, _memoryManager, dictionaryColumnSize,
                Math.min(_statsHistory.getEstimatedCardinality(column), _capacity), allocationContext);
1...676869...160Latest