Hi everyone im building a health and wellness start up and at the core is vector databses (pinecone) would really like to chat to someone who has a lot of knowldge of how to best start as I am re starting my project as my old one was corrupted with to many bugs

hey we are happy to chat. what happened with the old project? and what are you trying to do?

Old project is still this one i am talking about just coming across new obstacles and hurdles that I am trying to figure out

Hey @calm-arm-60496 - All great questions! Can you share more specifics about your use case? You mentioned "health and wellness" but it would be helpful to understand more about what your users need to accomplish, what your data looks like (what is the data, what format, etc) and the types of queries you expect to see. Additionally, would be helpful to understand what's not working for you today. Are you seeing misses on queries? - Contextualized chunking - there is some overhead here, so it doesn't always make sense. It might make sense if your data references other parts of data and only makes sense in the context of those other parts (think legal docs). - I assume for tags, you're talking metadata? - This will depend on what your data looks like and if it already has structured patterns that you need to match on (maybe a date separates a day's worth of data but then chunk the day's data using a different strategy). Again, would be helpful to better understand the issues you are seeing and understand your specific use case/data.

The biggest problem i have is that i know my corpus is correctly inputted in the vector store, thus pinecone has all the correct knowldge but getting that knowledge out of pinecone (in the correct manner) is what is causing me the biggest problems I am consistently iterating to try and find out if its the way I am querying the index or if was the way I chunked the data (not having contextualised chunks) the raw info is research papers from pubmed put in Pinecone Yes i was refering to meta tags, is there an optimal amount, like could I have to many?

Re: metadata, it really depends on what you're doing with it. We support up to 40kb of metadata and you can use this to filter/fetch data based on a piece of metadata (think team, category, etc.). The one limit I know people have run into is doing a

$in

filter with too many values as this can impact performance. Are you doing metadata filtering or are you just storing additional data and wondering if that's impacting your results?