Hi there! I work for a team that is currently using Azure Files as the system of record (everything in 1 file share), and they want to build a company-wide RAG chatbot application that uses Pinecone as the vector database. All the files in the file share have very specific permissions attached to them in terms of who in the company is allowed to see them. I was wondering, does Pinecone (in the enterprise plan, or any plan) have a way of honoring those permissions when an employee wants to chat with the company chatbot? Aside from being in different departments altogether (something that would seemingly be easily handled by Pinecone namespaces), if 2 people are on the same legal team, let’s say, but one employee is more senior, could the context for the more senior employee’s chatbot experience not be as restricted? Does Pinecone account for that when retrieving context, or is there a non-trivial way to configure it to account for that by understanding the file permissions? I might be asking this question horribly, so if I'm not making sense, I'm happy to explain further!

Hi @boundless-dentist-29324 - We don't have out of the box support for managing permissions of/access to files/chunks, but you can integrate your own authorization layer to determine who/what role has access. Here's a blog post about doing this with SpiceDB. You don't have to use that, but the approach is the same for other providers. Essentially you're storing a bit of information in metadata on what access is required and using the external authorization later to determine what that person is authorized for. Then using metadata filtering to only bring back the files/chunks they can access. https://www.pinecone.io/learn/rag-access-control/

This is super helpful, thank you so much!

Of course! Let us know if you have more questions as you get into it.

Sorry Jenna one more question haha. Are there plans to support Azure Files or other file-based storage systems, or will Pinecone integrations remain limited to object storage like S3 and Blob? I see under storage integrations in the Pinecone console, it is only Object Storage options from the cloud providers.

No worries! Questions are good! I have some too 🙂 1. Are you currently/planning to use the vector database? Or Assistant? 2. Are you expecting to be able to import raw files (i.e. a Word doc or a PDF) or are these files on Azure Files already chunked and vectorized?

Haha thank you Jenna! Nothing is actively being used yet (except Azure Files), but the plan is to use LangChain and Pinecone to build a RAG instance (the team is set on using LangChain for the orchestration). I’m not as familiar with Pinecone Assistant, but our current architecture assumes that the files will be enumerated, parsed, chunked, and vectorized by LangChain first, with Pinecone acting purely as the vector database to store and retrieve those embeddings. If you think Pinecone Assistant would be better than Langchain + Pinecone, or if any features of Pinecone Assistant or just the Pinecone vector database would interoperate well with Azure Files and Langchain, the team would definitely be open to hearing more about it!

Hi @boundless-dentist-29324 , apologies for the delay here. Pinecone Assistant is designed to take in raw files directly and performs an extensive amount of knowledge extraction (parsing, chunking, knowledge graph) and then also processes a query coming in to identify the right routing and chunk expansion necessary to give you the best context (or generative chat). The main goal of Pinecone Assistant is to leverage all of the best practices that we've built up over the years so you don't have to start from scratch with parsing, chunking, etc. We're seeing very high quality responses and would be eager to hear your feedback. We're just now beginning to roll out "data connectors" which will allow you to sync/stream data into the Assistant. Would be happy to connect you with our Product and Eng teams to learn more about what you're working on and share our roadmap as well.

Hi Perry! I would love that! Thank you so much!

hey, @boundless-dentist-29324 u’re asking it the right way .. this is actually one of the harder parts of building RAG systems in real orgs short answer: pinecone doesn’t natively “understand” ur file permissions. namespaces can separate broad groups, but they won’t handle fine-grained access like senior vs junior within the same team what usually works in practice is enforcing permissions before retrieval, not relying on the vector db itself. so instead of querying everything, you: * filter documents based on the user’s access rights (from your source of truth, like azure) * only embed/index metadata that includes those permissions * apply that filter at query time so the retriever never even sees restricted docs for more complex cases (like hierarchy within a team), you end up layering a permission service or policy check on top of retrieval, sometimes even doing a second pass to validate the final context before it goes to the model i’ve been building similar multi-agent / rag systems, and access control + leakage prevention is usually where most of the real complexity lives happy to go deeper if you want, this is a fun problem 👍