Hey @Andy Yeo (Airbyte) Not sure if this is the right place to ask this question, but i want to know if Asana connector will be upgraded or supported in future (like if getting data incrementally will be added as new feature). Appreciate your response.

Hi there. I am using AIRBYTE to transfer data from Mongo DB to Snowflake (edited) I created source connection with Mongo DB successfully. But as I go to use this source connection with destination connection (snowflake), it stops the service of my Mongo DB. Please help. (edited)

Why mysql to bigquery, update and delete cannot be incremented?

mysql to bigquery, what is the principle of increment

Hi all, I wrote an interesting article on integrating Airbyte into Prefect: https://medium.com/@adebayoadejare/modern-data-orchestration-stack-with-prefect-2-0-airbyte-and-dbt-e7c0e9b27add

good morning guys, I'm trying to connect the airbyte(local) with the minio(local - K3S) but the message "Please wait a little bit more…" stays forever

A desktop app for Airbyte. Fun! https://webcatalog.io/apps/airbyte/

Hey folks, I'm researching the space for the post-cookie web analytics. How do you feel the web analytics will look like in the modern data stack times?

I've been having some issues access files added to my S3 buckets by Airbyte and was curious if anyone has had a similar issue (or a solution). We use Deepnote for some data work and they have an integration with S3. It can read the bucket, see all the folders and explore them. But it shows no files in the folder and acts as if they are empty. If I create a test bucket and manually add files, there are no issues... Deepnote can't find anything and recommended me dig into Airbyte to see if there's something I've misconfigured or set up incorrectly. I can't find anything... so, last ditch effort is to ask here 😅

https://jsoncrack.com/editor amazing tool to see json schemas 🙂

Hi I hope it's ok posting here 🙏 CoRise is launching a FREE live, cohort-based, 2-week SQL Crash Course, taught by Sourabh Bajaj, co-founder and CTO of CoRise, ex Eng lead at Neeva and Tech Lead at Google. This course is perfect for anyone looking to start a career in data science or analytics. You'll learn what SQL is, why it's powerful, and how to get started writing queries. Join hundreds of learners from around the world in this highly interactive, engaging, and project driven learning experience. If you're interested enroll here

Hey! Is there an estimated date for release of V2 state message (I need per-stream states)? https://docs.airbyte.com/understanding-airbyte/airbyte-protocol#v2-coming-soon

Hi , During ingestion from source to destination. Is it possible that I can ingest only few columns. For Example in my source table there are over 200 columns. However, during ingestion I just want only selective ones. As In pipeline UI I can't see this option.

Hi again, can anyone give some hints how exactly the data-type binding works between different connectors ? for example the source stream i use is Oracle by default some columns have datatypes of VARCHAR and INTEGER. And The destination is Postgres by default the datatype for majority of columns by AirByte are TEXT and FLOAT. Can we actually change this to VARCHAR (255) and Int8 etc ??. I tried changing it via Octavia-CLI in the stream information, (TEXT -> VARCHAR) but it overwrites to jsonb object.

Hello, Jira to snowflake connection is stuck at waiting for logs… I’m at the latest connectors for both jira and snowflake. logs doesn’t show anything relevant; any pointers?

new airbyte website! 🙂 wdyt? kudos to @Talia Moyal for leading the redesign!

Complex data workflows need orchestration tools such as airflow to create, maintain, schedule and monitor the data pipelines. While airflow helps streamline these steps, troubleshooting a failed DAG can be a mind-numbing task - Digging through the logs, re-tracing the data transformation in each of the task instances and re-running the dags. Now, imagine the challenges of troubleshooting your dags that have 10s and 100s if not 1000s of tasks. While orchestration will bring order into the execution of the pipeline, ensuring the quality of orchestrated data pipelines is still a challenge. Join us on Wednesday to a webinar by lakeFS team to learn how to ensure data quality in orchestrated pipelines, and how to achieve full reproducibility of those processes. https://go.lakefs.io/troubleshoot-airflow-with-lakeFS

JFYI to anyone who works or writes stuff for GHA(Github Actions): https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

hey folks! Chris here, getting ready for Coalesce in New Orleans next week, can't wait! If you're in the area, we want to meet you! Please stop by our booth and say hi 👋🏼 to get an early adopter sticker and/or attend one of the happy hours we are planning! To get your sticker just come by our booth and tell us you are a user! Monday, Oct 17, 2022 - Flights & Bytes Wednesday, Oct 19, 2022 - Spirits in the Speakeasy

Unicorn Startup CTO Tells-all on How to Get the Most Out of the Cloud! Insurtech company Branch just became a Unicorn, and Joe Emison – their CTO, serial entrepreneur, and AWS Serverless Hero, will share insights into his success with the Serverless Toronto community. Join us (online) tomorrow to learn how to get your Tech Stack and Mindset right so that you can deliver software faster, better, and at a lower cost. Joe will also give you a sneak peek inside his upcoming book “Serverless as a Game Changer: How to Get the Most Out of the Cloud” so you can start applying his “formulas” for success next week – not in March 2023 when the rest of the world will be able to read it. Get your free Ticket now, and bring along plenty of questions: https://www.meetup.com/serverless-toronto/events/286456937/. We're looking forward to it being a fun-filled evening :)

Curious about startups, open source and identity? Here is a recent event at the DataTalks.Club where Sonal Goyal, founder of Zingg talks about her founder journey in open source while building identity resolution software.

How can you build an analytics practice? This was the question that Brady Gillerlain faced when he joined real-estate startup Ylopo. The company wanted to be more data-driven, but didn't know where to start. Join this conversation next week with Brady as he shares the lessons learned on his journey to build a robust analytics strategy. And we'll have him explain where the name Ylopo comes from 😉 Register here if you're interested

Heard about identity and entity resolution but not sure what they mean? Here is a primer on this topic by DataBeats. https://databeats.community/p/entity-resolution-sonal-goyal#details

It's exciting to see this PR merged! https://github.com/airbytehq/airbyte/pull/18031. Thank you @Sherif Nada! 👋waving lakefs

Composable CDPs - anyone building around stuff mentioned here? https://www.netspring.io/blog/composable-cdp-product-analytics/

Hi all! The Dagster team has recently updated our existing Airbyte integration to be fully compatible with the Dagster framework out-of-the-box. We created a new channel, #C048237B430, with integration development being led by our very own @Ben Pankow! Here’s some of his recent efforts in making the integration software-defined asset compatible. We can generate the corresponding assets in Dagster as you manage your connections in Airbyte. It’s pure magic ✨ If you’re interested in using this integration, join the channel to receive weekly updates, read our newly released guides, and discuss your use-cases airbyte

Hey folks, for those in Singapore, I’m going to be doing a talk and live demo on airbyte, dagster, dbt and Snowflake at Rakuten’s offices on November 24th 2022. Hope to connect with you there 🙂 https://www.meetup.com/data-engineering-sg/events/289448824/ Here’s a sneak peek of the slides: https://bit.ly/dataengineercamp-modern-elt-demo

Hi folks! We’ve just locked down the full schedule for this year’s Open Source Analytics Conference – OSA Con 2022 – and what a schedule it’s turned out to be! 20+ Talks 30+ Speakers 15 Community Sponsors (including Airbyte :D) 2 Keynotes, 2 Panel discussions, 2 Tracks on the same day! Thanks to the many people who have already signed up to join us on November 15th – and if you haven’t had a chance to register yet, please secure your spot for free as soon as possible!

💡 Virtual Talk: Level Up Your Data Lake - To ML and Beyond 🕑 Date: Today. 3rd Nov. 2:30 PM PST 🎯 Who is it for: Data Engineers, MLOps Engineers, ML Engineers, Data Practitioners ✅ Why is it relevant: If you work with big data, data lake or data pipelines, you will learn how to efficiently perform storage improvements on your data lake. By leveraging different file formats, table formats and data version control on your data lake, you can truly level it up for ML or Data Engineering workloads. 💸💸 Price: Register for FREE! Discount Code: FTSPKOPEN 🎤 Talk Link: https://go.lakefs.io/level-up-your-data-lake-w-lakeFS