Hi guys, is there any sort of "model card" for pinecone-sparse-english-v0? Really I'm interested in its tokenzer vocabulary size.

You mean this https://share.google/vopY10TOD01HX2R82 ? Or do you need more detailed information?

Hi @incalculable-balloon-29330 , the sparse-english model uses a bert-base-uncased tokenizer under the hood with vocab size 30,522. (https://huggingface.co/google-bert/bert-base-uncased/blob/main/tokenizer.json)](https://huggingface.co/google-bert/bert-base-uncased/blob/main/config.json)

Oh thanks guys! Just a followup, what about the output dimension?

there's not a normal dimension size to sparse embeddings like there is for dense vector embeddings (where output is D - floating point numbers) sparse embeddings look like this, where generally there's an (index, value) pair for each word, where the value is a relevance score for the word within the passage this is good reading on the perks of using sparse with traditional dense vectors: https://www.pinecone.io/blog/cascading-retrieval/

from pinecone import Pinecone

pc = Pinecone("API-KEY")

pc.inference.embed(
    model="pinecone-sparse-english-v0",
    inputs=["what is NVIDIA share price"],
    parameters={
    "input_type": "passage", # or query
    "return_tokens": True,
    }
)

# Response
...
{
    "vector_type": "sparse",
    "sparse_values": [0, 2.2207031, 5.3007812, 0, 0.32226562],
    "sparse_indices": [767227209, 1181836714, 1639424278, 2021799277, 3888873998],
    "sparse_tokens": ["what", "share", "nvidia", "is", "price"]
}

Ah yep, think where I'm getting at is whether there's some maximal index value, which could be needed if I end up converting these to scipy sparse format