Hi Team, did anyone try vector-index column with Pinot 1.4.0? I'm stuck for the last 3 days with the problem of getting zero records if I use vector-index column. I see that the Metadata API is also not returning anything about vector-index which makes me suspect that the vector indexing itselt is not happening. I have tried both modern indexes config and legacy properties config and both didn't help. For legacy config I see that pinot is changing my table config automatically to something like below.

"fieldConfigList": [
      {
        "name": "embedding",
        "encodingType": "RAW",
        "indexType": "VECTOR",
        "indexTypes": [
          "VECTOR"
        ],
        "indexes": null,
        "properties": {
          "vectorIndexType": "HNSW",
          "vectorDimension": "64",
          "vectorDistanceFunction": "INNER_PRODUCT",
          "version": "1"
        },
        "tierOverwrites": null
      }
    ]

I have tried reloading all the segments everytime I change the config but the metadata API never returned something about vector-index for that column. Can someone help me with this please?

how is your embedding column defined in the schema and what's the data look like

Thank you for the reply. This is my embedding column:

{
  "name": "embedding",
  "dataType": "FLOAT",
  "fieldType": "DIMENSION",
  "singleValueField": false
}

This is how my data looks, not able to get the entire value in the screenshot but it's a 64 length array. Let me know if anymore info helps.

this definition seems to be ok, what's your query ? is it something like

where VECTOR_SIMILARITY(embedding, [search embedding], 5)

yes, this is my sample query i'm using for testing

WHERE VECTOR_SIMILARITY(embedding, ARRAY[-0.07831726, -0.2119323, 0.11863201, 0.069690615, 0.2813305, -0.02584411, -0.117572516, 0.013397558, 0.07510444, -0.08048767, 0.0898106, 0.083453424, 0.037585236, 0.0655264, -0.13094723, -0.08184803, -0.06359755, -0.05847629, 0.008707356, 0.21086763, -0.10072265, -0.047040176, -0.17444366, -0.050620943, 0.23826164, -0.010716335, 0.08178077, 0.22486627, -0.0033848207, -0.061445918, -0.054641392, 0.19359636, 0.24487428, -0.031223591, -0.0058864253, 0.16827327, 0.014289394, -0.13401766, 0.20543532, -0.074380636, 0.03430772, -0.09490038, 0.10463648, -0.09886578, 0.10548536, 0.1929114, -0.05888739, 0.055344768, -0.042525716, 0.14039323, -0.019105438, 0.14100789, 0.14475566, -0.0066321488, -0.15118349, -0.01205526, 0.05960668, -0.098317765, 0.028235469, 0.14057179, -0.13916609, 0.3174897, -0.12983285, 0.16192645], 10)

I don't know if it's useful but these are the stats when I run the query

the stats show no segments matched then it means no data are returned.

I see the below error (number of docs=0) with only ONE segment when I load the segments into the table. However, I see on the UI that the same segment has 42000 docs and even manually I confirmed that the embedding column has no NULL values and it's not empty. Do you know the possible reason for this error in this case? and why is it happening only with one segment? why is the indexing not happening with the other 24 segments which didn't have this error log?

hmm, this means the index build has issue.

1. I tried a foreceful reload (download) from swagger UI 2. I also tried deleting the table, schema and pinot data. Created a new table and loaded data again I still see the same error about maxDoc=0. Any queries other than vector_similarity is returning rows so the data is not empty for sure. The metadata also doesn't have anything about vector-index.

is it just one segment or all the segment? can you DM me some sample data and schema/table config so I can test ?

during reloading of segments I saw that multiple segments got this error. Ok, I will DM you the details.