Hi Everyone octavia wave, As you might know by now, Airbyte is now able to write both structured and unstructured data to Vector Databases, in addition to working on deeper integration with Langchain and LlamaIndex 🚀🚀 This is a big step forward as it unlocks a lot of use cases for Airbyte! For instance, we just built a “connector development support” AI chat bot, using Airbyte, Langchain, Pinecone and OpenAI. We decided to create a tutorial to show you how we did it 🔥. By the end of it, you will know how to: • Extract unstructured data from a variety of sources using Airbyte • Use Airbyte to efficiently load data into a vector database, preparing the data for LLM usage along the way • Integrate a vector database into your LLM to ask questions about your proprietary data If you’re interested in AI-powered use cases, we’re doing a live demo of moving data to a vector database and creating a Slack Support bot, click here to join!

Amazing! Will it be possible to use different LLM's like LLaMA(2) in the future as well? To embed the data without having to use OpenAI?

That's very cool, however connector requires a paid subscription for Pinecore?

Hi @Chris, Yes, you will be able to, as long as they are publicly hosted/available via API.

@Renat Zubayrov Thanks for reporting, I'm looking into this problem (created an issue here: https://github.com/airbytehq/airbyte/issues/29356 ) In short - the behavior of Pinecone starter pods changed slightly and we need to adjust the way our connector works. I'll be working on having a fix up quickly. For now, you can set the sync mode to

Incremental | Append

and it should work.

thanks