This week, I have a few amazing people in mind that I wanted to recognize octavia loves airbyte heart for the most helpful contributors: @Oleksandr Bazarnov [GL], @Jeff Crooks and @Yaroslav Dudar 🔥 for the contributors that sparked the most discussion: @Jarrod Parkes, @Jiyuan Zheng and @Mohammad Safari So thank you!!

As usual, it's patch notes time <!channel>! We're going over the changes from

0.29.17

and before... and we're doing things a little differently this week, as there are a LOT of big updates that we want you to focus on: all the bugfixes will be attached in the thread on this announcement, so open the thread if you want to see them. Without further ado... New Source: Facebook Pages New Destination: MongoDB New Destination: DynamoDB • 💎 You can now send notifications via webhook for successes and failures on Airbyte syncs. This means that you can now set up external apps that let you know about the status of your Airbyte syncs without any hassle. (This is a massive contribution by @Pras, thank you) 🎉 • 💎 Scheduling jobs and worker jobs are now separated, allowing for workers to be scaled horizontally. To be specific, this means that you can now orchestrate thousands of simultaneous jobs on K8s if you so desire (make sure to update your docker-compose file for this to work). • 🎉 When developing a connector, you can now preview what your spec looks like in real time with this process. • 🎉 Oracle destination: Now has basic normalization. • 🎉  Added XLSB (binary excel) support to the Files source (contributed by Muutech). • 🎉 You can now properly cancel K8s deployments. • ✨ S3 source: Support for Parquet format. • ✨  Github source: Branches, repositories, organization users, tags, and pull request stats streams added (contributed by @Christopher Wu). • ✨ BigQuery destination: Added GCS upload option. • ✨ Salesforce source: Now Airbyte native. • ✨ Redshift destination: Optimized for performance. • 🏗️ CDK: 💎 We’ve released a tool to generate JSON Schemas from OpenAPI specs. This should make specifying schemas for API connectors a breeze! 💎 • 🏗️ CDK: Source Acceptance Tests now verify that connectors correctly format strings which are declared as using 

date-time

 and 

date

 formats. • 🏗️ CDK: Add private options to help in testing: 

_limit

 and 

_page_size

 are now accepted by any CDK connector to minimze your output size for quick iteration while testing. As usual... thank you to our wonderful contributors this week: @Pras, @Christopher Wu, @Brian M, @yahu98, @zuc, @jinnig, and @luizgribeiro!

This week, I have a few amazing people in mind that I wanted to recognize octavia loves airbyte heart for the most helpful contributors: @Jeff Crooks, @Iryna Kruk and @Oleksandr Bazarnov [GL] 🔥 for the contributors that sparked the most discussion: @Lior Halperin, @Osinachi Chukwujama and @Jiyuan Zheng So thank you!!

Welcome @Adam Knobel, Airbyte's new Head of Sales!!

This week in our demo hour, we'll go over how to set up Airbyte and Google Composer on GCP and send data from a Postgres Database to BigQuery! airbyte rocket If you're curious on how to leverage Airbyte with Airflow this is your chance! You can register to the Livestorm session here.

We are launching our Connector Marketplace 🎉 <!channel>! (along with our Discourse forum). What does this mean? It's a place for people to commission the construction of Airbyte connectors. Community members that are skilled at building connectors can meet with teams and companies that need connectors built to form a mutually beneficial relationship. Check the thread for more details!

Now let's get to the

0.29.19

changelog. As with last time, bug fixes are in the thread! New Destination: Databricks 🎉 New Source: Google Search Console New Source: Close.com 🏗️ Python CDK: Now supports auth workflows involving query params. 🏗️ Java CDK: You can now run the connector gradle build script on Macs with M1 chips! (contributed by @Harsha Teja Kanna) 💎 Google Ads source: You can now specify user-specified queries in GAQL. ✨ GitHub source: All streams with a parent stream use cached parent stream data when possible. ✨ Shopify source: Substantial performance improvements to the incremental sync mode. ✨ Stripe source: Now supports the

PaymentIntents

stream. ✨ Pipedrive source: Now supports the

Organizations

stream. ✨ Sendgrid source: Now supports the

SingleSendStats

stream. ✨ Bing Ads source: Now supports the

Report

stream. ✨ GitHub source: Now supports the

Reactions

stream. ✨ MongoDB source: Now Airbyte native! Also, a quick shout out to @Jinni Gu and their team who made the DynamoDB destination that we announced last week! As usual, thank you to all of our contributors: @Harsha Teja Kanna, @Manav, @Maciej Nędza, @mauro, @Brian M, @Iakov Salikov, @Eliziario (Marcos Santos), @coeurdestenebres, and @mohammadbolt

Hey community! We already have quite a few Octavia emojis octavia loves octavia rocket octavia partying octavia thinking 👋 octavia thanks 🤔 octavia rolling eyes octavia shock. are they any Octavia emojis you'd like to see 🙂 ?

Hello everyone! Please welcome @Harshith (Airbyte) who has officially joined the Airbyte team today as our User Success Engineer. We're super excited to have him on board, especially since he has been a valuable member of our community for a while now!

If anyone has any more questions for @Tim Shea, place them in this thread!

Hey everyone 👋 ! We just wanted to share a few milestones our community achieved this week! • airbyte star 4k stars on GitHub • S 3k Slack community members • airbyte growth  100B records replicated in less than a month • airbyte rocket  1.8k deployments per month (from 300 back 6 months ago, so 6x in 6 months) We are honored to be working with such an amazing community and together will achieve our long-term vision to commoditize data integration. Thank you all so much 🙏!

It's patch notes time <!channel>, let's go over the changes from our new minor version,

v0.30.0

. As usual, bug fixes are in the thread. New source: LinkedIn Ads New source: Kafka New source: Lever Hiring 🎉 New License: Nothing changes for users of Airbyte/contributors. You just can't sell your own Airbyte Cloud 😁 💎 New API endpoint: You can now call

connections/search

in the web backend API to search sources and destinations. (contributed by @Mario Molina) 💎 K8s: Added support for

ImagePullSecrets

for connector images. 💎 MSSQL, Oracle, MySQL sources & destinations: Now support connection via SSH (Bastion server) ✨ MySQL destination: Now supports connection via TLS/SSL ✨ BigQuery (denormalized) destination: Supports reading BigQuery types such as

date

by reading the

format

field (contributed by @Nicolas Moreau) ✨ Hubspot source: Added contacts associations to the

deals

stream. ✨ GitHub source: Now supports pulling commits from user-specified branches. ✨ Google Search Console source: Now accepts admin email as input when using a service account key. ✨ Greenhouse source: Now identifies API streams it has access to if permissions are limited. ✨ Marketo source: Now Airbyte native. ✨ S3 source: Now supports any source that conforms to the S3 protocol (Non-AWS S3). ✨ Shopify source: Now reports

pre_tax_price

on the

line_items

stream if you have Shopify Plus. ✨ Stripe source: Now actually uses the mandatory

start_date

config field for incremental syncs. 🏗️ Python CDK: Now supports passing custom headers to the requests in OAuth2, enabling token refresh calls. 🏗️ Python CDK: Parent streams can now be configured to cache data for their child streams. 🏗️ Python CDK: Now has a

Transformer

class that can cast record fields to the data type expected by the schema. Thank you to our awesome contributors. Y'all are amazing: @Mario Molina, @Pras, @Vladimir Remar, @Christopher Wu, @gunu, @Juliano Benvenuto Piovezan, @Brian M, @Justinas Lukasevicius, @Jonathan Stacks, @Christian Gagnon, @Nicolas Moreau, @aristidednd, @camro, @minimax75, @peter-mcconnell, and @sashkalife!

And... we'd like to announce that @Jonathan Stacks is our community MVP for September! His work on the Airbyte Helm chart (which can be found here) is massive and will help our community deploy Airbyte on Kubernetes. Congrats Jon!

Hi <!channel>! I just wanted to let everyone know that we will be starting Airbyte Hacktoberfest next week on 10/11, lasting until 11/19. We're having a contest to see who can build the best and most connectors, so we want to encourage everyone to try their hand out at using our CDK! There's a lot of integrations that would be cool to see come to life over the next month. We will be awarding $500 for each connector (no limit!) built during this time and we'll be giving out $1000 and $750, respectively, to the two individuals that create the highest quality and most connectors during this time. We'll also be sending lots of swag to creators of fun, creative, spooky, and technically impressive connectors. All the details and resources can be found at the GitHub repo for the event. Make sure to follow these guidelines before starting an entry. If there are any questions or if you run into any roadblocks, we can help you out in #C01VDDEGL7M. I look forward to coming together as a community to teach each other about connector development and making some cool integrations!

Happy Monday! Please welcome @Benoit Moriceau (Airbyte) who's joining our team today as a Senior Software Engineer and @Brian Cannon who's joining our team today as our Technical Recruiter Lead!! party parrotoctavia loves

Hey <!channel>! It's patch notes time. Let's go over the changes from

0.30.24

and before. But before we do, let's get a quick update on how Hacktober is going! 🎃 Hacktoberfest Submissions 🎃 ----------------------------------------- New Destination: Elasticsearch (contributed by @Jeremy Branham) New Source: Salesloft (contributed by @Pras) New Source: OneSignal (contributed by @Bo) New Source: Strava (contributed by @terencecho) New Source: Lemlist (contributed by @Igli Koxha) New Source: Amazon SQS (contributed by @Alasdair Brown) New Source: Freshservices (contributed by @Tuan Nguyen) New Source: Freshsales (contributed by @Tuan Nguyen) New Source: Appsflyer (contributed by @Achmad Syarif Hidayatullah) New Source: Paystack (contributed by @Foluso Ogunlana) New Source: Sentry (contributed by @koji matsumoto) New Source: Retently (contributed by @Subhash Gopalakrishnan) New Source: Delighted! (contributed by @Rodrigo Parra) ----------------------------------------- with 18 more currently in review... ----------------------------------------- 🎉 Incremental Normalization is here! 🎉 💎 Basic normalization no longer runs on already normalized data, making it way faster and cheaper. 💎 ----------------------------------------- 🎉 Airbyte Compiles on M1 Macs! Airbyte developers with M1 chips in their MacBooks can now compile the project and run the server. This is a major step towards being able to fully run Airbyte on M1. (contributed by @Harsha Teja Kanna) ✨ BigQuery Destination: You can now run transformations in batches, preventing queries from hitting BigQuery limits. (contributed by @Andrés Bravo) ✨ S3 Source: Memory and Performance optimizations, also some fancy new PyArrow CSV configuration options. ✨ Zuora Source: Now supports

Unlimited

as an option for the Data Query Live API. ✨ Clickhouse Source: Now supports SSL and connection via SSH tunneling. As usual, thank you to our awesome contributors that have done awesome work during this productive spooky season: @Tuan Nguyen, @Achmad Syarif Hidayatullah, @Christopher Wu, @Andrés Bravo, @Harsha Teja Kanna, @Collin Scangarella, @haoranyu, @koji matsumoto, @Subhash Gopalakrishnan, @Jeremy Branham, @Rodrigo Parra, @Foluso Ogunlana, @EdBizarro, @Gergely Lendvai, @Rodeoclash, @terencecho, @Igli Koxha, @Alasdair Brown, @bbugh, @Pras, @Bo, @Xiangxuan Liu, @Hai To, @s-mawjee, @Mario Molina, @SamyPesse, @Yuhui Shi, @Maciej Nędza, @Matt Hoag, and @denis-sokolov!

Hey <!channel>! Just wanted to give everyone a heads up about potential issues with your first syncs with our new Incremental Normalization feature. ⚠️ If you are running into issues, make sure to delete your old normalized tables before running the new version of normalization (we've noticed this mostly on BigQuery destinations). ℹ️ If you want to re-partition your BigQuery data if you don't like the way we do it, you can refer to this recipe to tune it your liking.

Hey <!channel>! We've been seeing some awesome work on the Airbyte repo with Hacktober... here are the current stats: • 50 total connectors are either being worked on or are completed. • 33 PRs have been submitted to the Airbyte repo for Hacktober, with 14 of them merged! • @Tuan Nguyen is currently in the lead with the most connectors merged at 3, and he also has 3 in review! To help get everyone across the finish line, we wanted to showcase some of our community connector builders that are participating, to give some development insights and tips for connector building. ------------------------------------------------------------------------------------------------- So welcome to our Airbyte Community Call Series: _Connector Development Edition!_ @Luis Gomez from Calixa will be presenting next week on 11/9. Register here! @Apostol Tegko and @Peter Varga from Fnatic will be presenting the week after on 11/16. Register here! So definitely join if any of the following apply: • You want to start building a connector and don't know where to start. • You are currently building a connector and are stuck. • You want to learn some cool tips about connector building. We look forward to seeing you there!

Please welcome @Erica Struthers who is joining our team as Head of Technical Support and @Edward Farraye who is joining our team as Head of Demand Gen today! airbyte rocket

Hi <!channel>! Just a quick reminder that our community call will be starting in just under an hour. Register here and I look forward to seeing you all there!

Hi <!channel>, time to go over changes from

v.30.39

! And... let's get another update on Hacktoberfest. 🎃 Hacktoberfest Submissions 🎃 ----------------------------------------- New Destination: Cassandra (contributed by @Ivica Taseski) New Destination: Pulsar (contributed by @Mario Molina) New Source: Confluence (contributed by @Tuan Nguyen) New Source: Monday (contributed by @Tuan Nguyen) New Source: Commerce Tools (contributed by @James Wilson) New Source: Pinterest Marketing (contributed by us octavia loves) ----------------------------------------- ✨ Shopify Source: Now supports the

FulfillmentOrders

and

Fulfillments

streams. ✨ Greenhouse Source: Now supports the

Demographics

stream. ✨ Recharge Source: Broken requests should now be re-requested with improved backoff. ✨ Stripe Source: Now supports the

checkout_sessions

,

checkout_sessions_line_item

, and

promotion_codes

streams. ✨ Db2 Source: Now supports SSL. 🐛 We've made some updates to incremental normalization to fix some outstanding issues. [Details] 🐛 Airbyte Server no longer crashes due to too many open files. 🐛 MSSQL Source: Data type conversion with smalldatetime and smallmoney works correctly now. 🐛 Salesforce Source:

anyType

fields can now be retrieved properly with the BULK API 🐛 BigQuery-Denormalized Destination: Fixed JSON parsing with

$ref

fields. As usual, thank you to our awesome contributors that have done awesome work during the last week: @Tuan Nguyen, @Harsha Teja Kanna, @Aaditya S, @James Wilson, @Vladimir Remar, @Yuhui Shi, @Mario Molina, @Ivica Taseski, @Collin Scangarella, and @haoranyu!

Hey everyone 👋 ! We've been iterating lately on how we could improve the connection settings. Our goals were to: • make the first-time user experience easier with a simpler onboarding • enable batch editing of streams in a much easier and self-explanatory way • restructure the connection detail pages to make more sense You can see the *designs *here.  Don't hesitate to add your comments. If you want to play with a mockup prototype, here's one. Looking forward to getting your feedback!

Please welcome @Andy Yeo (Airbyte) who is officially joining us today as our Sr PM, Connector / Extensibility from San Francisco! 🚀

Hey everyone! It is my pleasure to announce that @Tuan Nguyen is the Airbyte MVP for October! Tuan has accomplished some amazing feats in our community that we'd love to recognize. Please join me in celebrating his accomplishments! Tuan has: • Contributed four connectors to Airbyte Open-Source during Hacktober. • Published an awesome piece of content with us about partitioning your BigQuery data. • Gave a at an Airbyte Community Call about how he uses Airbyte at Joon Solutions.

Hey <!channel>! Let's go over all the changes from 

v.32.1

 and prior! But first, there's an important announcement I need to make about upgrading Airbyte to 

v.32.1

. ⚠️ WARNING ⚠️ ------------------------ Upgrading to

v.32.0

is equivalent to a major version bump. If your current version is < 

v.32.0

 , you must upgrade to 

v.32.0

 first before upgrading to any later version Keep in mind that this upgrade requires your all of your connector 

Specs

 to be retrievable, or Airbyte will fail on startup. You can force delete your connector 

Specs

 by setting the 

VERSION_0_32_0_FORCE_UPGRADE

 environment variable to 

true

. Steps to specifically check out 

v.32.0

 and details around this breaking change can be found here. ------------------------ Now back to our regularly scheduled programming. 🎃 Hacktoberfest Submissions 🎃 ----------------------------------------- New Destination: ScyllaDB (contributed by @Ivica Taseski) New Source: Azure Table Storage (contributed by @geekwhocodes) New Source: Linnworks (contributed by @Juozas) ----------------------------------------- ✨ Source MySQL: Now has basic performance tests. ✨ Source Salesforce: We now automatically transform and handle incorrect data for the

anyType

and

calculated

types. 🐛 IBM Db2 Source: Now handles conversion from

DECFLOAT

to

BigDecimal

correctly. 🐛 MSSQL Source: Now handles

VARBINARY

correctly. 🐛 CockroachDB Source: Improved parsing of various data types. As usual, thank you so much to our wonderful contributors this week that have made Airbyte into what it is today: @Achmad Syarif Hidayatullah, @Tuan Nguyen, @Ivica Taseski, @Hai To, @Juozas, @gunu, @Shadab, @Per-Victor Persson, and @Harsha Teja Kanna!

This week, I have a few amazing people in mind that I wanted to recognize octavia loves airbyte heart for the most helpful contributors: @Lake Mossman, @Benoit Moriceau (Airbyte) and @Jason Edwards 🔥 for the contributors that sparked the most discussion: @Matheus de Freitas Andrade, @Clovis Masson and @Sheshan Without all of you, the community wouldn't be the same (well, as far as last week goes 😉). So thank you!!

Please welcome @Edward Gao who's joining Airbyte as a Sr Software Engineer, Connectors today! airbyte rocket

Hey Airbyte Community! As per usual in this <!channel>, I will go over the patch notes from

v0.33.6

to

v0.32.9

! But before I do... Make sure to register for our community call next week. We'll be talking with the Prefect team about their brand new integration with Airbyte, allowing you to orchestrate and schedule all of your syncs! Sign up here. ⚠️ WARNING ⚠️ ------------------------ •

v0.33.0

is a minor version with breaking changes" -> "is a minor version which changes env var configuration names. If you have any custom configurations check the new values. •

v0.33.0

has a bug that affects GCS logs on Kubernetes. Upgrade straight to

v0.33.2

if you are running a K8s deployment of Airbyte. ------------------------ 🎉 New Source: Mailgun 🎉 Snowflake Destination: You can now stage your inserts, making them much faster. ✨ Google Ads Source: Source configuration is now more clear. ✨ Google Analytics Source: Source configuration is now more clear. ✨ S3 Destination: You can now write timestamps in Avro and Parquet formats. ✨ BigQuery & BigQuery Denormalized Destinations: Now use byte-based buffering for batch inserts. ✨ Iterable Source: Now has email validation on the

list_users

stream. 🐛 Incremental normalization now works properly with empty tables. 🐛 LinkedIn Ads Source:

429

response is now properly handled. 🐛 Intercom Source: Now handles failed pagination requests with backoffs. 🐛 Intercom Source: No longer drops records from the

conversation

stream. 🐛 Google Analytics Source:

400

errors no longer get ignored with custom reports. 🐛 Marketo Source: The

createdAt

and

updatedAt

fields are now formatted correctly. As usual, thank you to our awesome community contributors who contributed to the project in the past week: @Joël Luijmes, @Per-Victor Persson, @Ciaran O'Brien, @James Wilson, @Bruno Nirello, @gunu, @Antonio Grass, and @Deividas J!

Hey <!channel>! We have two events coming up in the next two days that I'd like to remind everyone about! • Tomorrow (12/14 @ 12:30 PM PT), Alexander Streed from Prefect will be presenting at our monthy Community Call, showing off the brand new integration with Airbyte, displaying how to orchestrate your Airbyte syncs with the Prefect Platform. Sign up here! • Day after tomorrow (12/15 @ 11:00 AM PT), I will be doing a demo at our Demo Hours displaying how easy it is to deploy Airbyte on Kubernetes using the open-source plural.sh platform. Sign up for that here! • Essa quarta-feira dia 15 as 19h BRT também acontecerá o meetup virtual da comunidade dbt no Brasil onde @[DEPRECATED] Marcos Marx irá falar sobre como construir um pipeline de dados moderno com Airbyte + dbt! Meetup em Português! Link para se cadastrar. We look forward to seeing all of you at the events leading up to the holidays! We'd like to also give the community a heads up that we are giving our awesome USE team a break for the following days and won't be providing active user support for these days to give them some time off for the holidays: • Friday, December 24th • Friday, December 31st We wish everyone an awesome holiday season!

<!channel> We are live with our Community Call announcing our integration with Prefect! Join here to view it live: https://app.livestorm.co/airbyte/airbyte-community-call-08-ft-alexander-streed-from-prefect?type=detailed