Hi, I am facing a very interesting problem with Pinecone (and just in general other Vector DBs). Is there anyone from the Pinecone development team here who I can reach out to regarding this problem? Happy to jump on a quick call too. - Shubham

Hi Shubham! Could you explain your problem here so we can better help? Also, if anyone else has the same problem as you, they can learn from your experiences too.

I'll explain high level but can provide more details in a private message if required. I am producing my entity embeddings using a weighted concatenation of multiple kinds of embeddings (text, numeric, etc) to capture multiple facets of an entity, and using either cosine/dot product depending on whether they're normalized at the end. FAISS FlatIP (exact search) gives me expected results (results for normalized and unnormalized are identical). Amazon S3 Vectors brings up a few weird, unexpected results in top K, and Pinecone results do not make sense at all.

I think I'm following. Are these embeddings literal features you've extracted using some other process, or through a model?

Text embeddings are from a pre-trained transformer model, numeric features are encoded using gaussian encoding, etc. So heterogenous processes.

pc.create_index(
        name=index_name,
        vector_type="dense",
        dimension=135,
        metric="cosine", # also tried dotproduct
        spec=ServerlessSpec(
            cloud="aws",
            region="us-east-1"
        ),
        deletion_protection="disabled",
        tags={
            "environment": "development"
        }
)

And when you say you are getting "expected results", what does this mean?

Okay, I'll provide these details to you in a private message.