Hey all, we keep running into problems enabling te...
# troubleshooting
j
Hey all, we keep running into problems enabling text search on a field in our cluster (running 1.2.0). The field basically contains search queries that users enter on our e-commerce website (so there are a lot of unique values and can also be somewhat long, I suppose). As soon as we enable text search, CPU on our server pods starts to grow until it maxes out and the pods die completely. Are there any metrics available that could give us better visibility into what might be happening? I don't see any text search-specific metrics in the documentation Or are there table configuration options we could try that might mitigate the issue? Thanks in advance for your help!
this is the related fragment of our table config
Copy code
"fieldConfigList": [
    {
      "name":"search_term",
      "encodingType":"RAW",
      "indexTypes":["TEXT"],
   }
  ],
x
when you say enable text index, did you reload the table or it’s just updating the table config and only new segments are building the text index
j
each time i update the table config, i reload all segments. but it also happened when we first deployed the table and it was just beginning to ingest data
x
got it, so i suppose your ingestion kafka is from the earliest offset right?
in this case, it’s likely due to the parallelism of segment reloading. As well as data consumption in the backfill mode.
text index build takes more cpu
what you can do is to limit the concurrent segment loading
j
ahh, that's really helpful -- thank you! would that be the
pinot.server.max.segment.preprocess.parallelism
config?
and are there a metrics I can watch to see the number of segments that get reloaded, or the number of concurrent segments that are reloaded?
x
yes, this config is for restart servers, there is also one more config to control the helix transitions processed at a time
actually what you found is correct this will limit the in parallel text build
For realtime table bootstrap, there is a config for max segment concurrent build
pinot.server.instance.realtime.max.parallel.segment.builds=2
j
ahh that looks super promising too, thank you! I'm trying mixtures of configs now and will follow up with how it goes