https://pinot.apache.org/ logo
Join Slack
Powered by
We are using pinot to store log data and noticed t...
# troubleshooting
e
We are using pinot to store log data and noticed that string columns are truncated at 512 characters. Is there another datatype or setting we should use to increase the length?
n
e
Thanks!
I guess the default is 512 for string?
Ah, I see it in FieldSpec
n
yes
Copy code
private static final int DEFAULT_MAX_LENGTH = 512;
// NOTE: for STRING column, this is the max number of characters; for BYTES column, this is the max number of bytes
  private int _maxLength = DEFAULT_MAX_LENGTH;
e
Would there be any issues with specifying a really large value? We are going to try out using the text index - using it to store log records
n
not sure. @Mayank @Jackie ?
e
We're trying 2mb out, we can let you know also 🙂
But if you have any serious reservations let me know
And thanks for your help again!
s
@Elon, are you going to use text_match?
e
Yep:)
s
There is a writer config that allows easy handling for large records (> 1MB per record)
I recommend enabling that
e
Oh nice! What's that?
s
What's the largest length (in terms of number of characters) for a cell in this column?
e
We're trying 250k characters
i.e. ~2mb
s
are they all standard ascii?
e
utf8
is that ok?
s
then fine. I asked because UTF-8 although takes 1 byte per character for standard ASCII, it also takes upto 4 bytes for higher range
so in that 250k characters kind of translate to 5MB
e
Makes sense
What's the magic config? 🙂
s
please enable the text index as follows
Copy code
"fieldConfigList": [
        {
          "encodingType": "RAW",
          "indexType": "TEXT",
          "name": "yourTextColumn",
          "properties": {
            "deriveNumDocsPerChunkForRawIndex": "true"
          }
        }
      ]

      "tableIndexConfig": {
        "noDictionaryColumns": [
          "yourTextColumn"
        ]
      }
Set this in table config
since the raw data is huge in size, I would not recommend creating dictionary on that
since it is essentially a log and you will be doing arbitrary text search as opposed to fixed/exact matches
e
Thanks, this is great!
s
The docs already talk about how to enable text index but since I added the writer to handle blob like text later, that is probably not in the docs. I will add
Please let me know if you have any questions
j
Is 
rawIndexWriterVersion
mandatory?
@Sidd Should we fix the version in the table config?
s
So using V3 rawIndexWriterVersion may not be necessary if the total size of raw data for text column isn't > 2GB
e
Does it hurt to include that though?
k
This is an amazing thread, please add this to docs
👍 1
e
You are all so helpful, this is incredible, everyone wants to use pinot here... for everything :)
🍷 2
s
So here is the thing: it doesn't hurt since it has been tested and that's what we are using at Li. In case there is some other bug (for some reason) and you ever decide to downgrade/rollback Pinot version, we will have a problem since V3 version uses 8 byte offsets and the prior version uses 4 byte. The writers are backward compatible but they aren't (and can't be) forward compatible. So that's why the rollback problem
@Elon, how many rows (roughly) per segment?
j
Are we planning to move to the new version in the new release?
e
We set the segment size at 200mb, and most of the log records are tiny, like 500 chars, 2kb per row so ~ 100k rows per segment, is that a good number?
I don't think we will ever roll back to a previous version, we will keep moving forward 🙂
s
Good to know. I just wanted to know if the total size is getting huge and you need the V3 writer. Use this config for now, push the data and let's test
e
Text index is working great. Before the text index was created each segment was 365mb, now it's 9-11mb and I looked at the metadata.properties files in each directory, each segment is about 100k rows
Is there a compaction job that runs?
s
There should be an increase in size
in the segment root directory, what does du -sh . indicate?
cd segment/v3, du -sh .
e
~11mb per segment
Just to be sure I can create another table without the text index to compare side by side
And why do the segments seem to be capped at 100k rows, atleast metadata.properties reports the highest rowcount at 100k exactly for all segments except the consuming one
s
yeah, you may want to look at the size of segment with and without text index.
For the offline segments, the number of rows per segment is dependent on the user's data prep job. So user has control how and what size (numrows) segments are created in offline flow
👍 1
e
For this table we only have realtime
n
regarding 100k rows cap, you likely have segment size based threshold enabled for creating segments in realtime. this happens if property “realtime.flush.threshold.size”=0. Pinot will slowly ramp up number of rows in the segment (with initial size 100k) until it gets to the desired size
e
Thanks! That's looks like exactly what's happening:)
We have this as the config for threasholds:
Copy code
"realtime.segment.flush.threshold.time": "6h",
        "realtime.segment.flush.threshold.size": "0",
        "realtime.segment.flush.desired.size": "200M",
👌 1
Open in Slack
PreviousNext
Powered by