Hey everyone <!channel>! :octavia-hello: This mon...
# announcements
j
Hey everyone <!channel>! 👋 This month, we released Postgres in GA! The team also published quite a lot of content on the blog and our Youtube channel, including some new Lakehouse guides and Dagster Day recaps. We are excited to be supporting and presenting at big upcoming industry events: • 21-22 Sept Big Data London • 26-27 Sept Big Data Paris • 17-21 Oct dbt Coalesce (New Orleans) • for San Francisco ◦ @Michel will be at State of Data Engineering on Sept 8! ◦ sign up to the Data Mash and Data Council SF meetups coming soon! • Not to forget - our own virtual move(data) conference Nov 8-10! ----- So what about our progress? In September, apart from Postgres GA we shipped many sources to Beta: • SurveyMonkey • Harvest • Zendesk Chat • File And an exciting new Destination: SQLite!!! With many more in alpha thanks to our awesome community (see thread) 👇 We are also releasing many big improvements to Airbyte, including features for normalization, observability, self hosting (check our Helm chart updates) and OAuth login! Check the thread to see all the patch notes, features and bug fixes, there are a lot of them! And don’t hesitate to ask questions there, we’ll be happy to answer 🙂 . Thank you to our awesome contributors! @Ryan (who recently joined us! welcome!), @Yuhui Shi (who contributed the SQLite connedestination!), @dtt101 (who contributed AppFollow), @Mauricio R S Jr (who contributed Freshcaller), @Jordan Scott (who contributed Linkedin Pages), @sivankumar, @Yiyang, @nipunraprashan (thanks for working on our Helm Chart), and many more!
🔥 10
octavia rocket 1
👋 2
this is fine fire 1
airbyte rocket 7
octavia wave 2
octavia partying 1
party parrot 4
yay 1
airbyte 2
airbyte code 1
octavia shock 1
airbyte growth 3
🎉 1
octavia thinking 1
keanu thanks 1
venting 1
octavia rolling eyes 1
🔥 6
Here are the patch notes from v0.39.42-alpha to v0.40.3: New and improved features • Airbyte is no longer tagged
-alpha
! (#15766) • New Destination: SQLite (#15018) • *Headline: Source Postgres is now in GA!*This means: ▪︎ We’ve verified a reliable experience for a (vast) majority of our customers using the connector ▪︎ Validated the UX and docs continue to be high quality and accessible ▪︎ Solved any outstanding bugs surfaced during the beta period ◦ Are now ready for prime time on this connector!! ◦ Updated Postgres doc (#15166) ◦ Align regular and CDC integration tests and data mappers; improve BCE date handling (#14534) ◦ emit state messages more frequently for incremental sync (#14903) ◦ sync data from beginning if lsn is no longer valid in cdc (#15077) ◦ Align Postgres Destination regular and strict encrypt versions (#15261) ◦ fix first record wait time parsing bug (#15273) ◦ fixed normalization java tests after changes in Python part (#15289) • New Sources and Promotions ◦ Source SurveyMonkey is now in Beta (#14998) • Source Harvest is now in Beta (#15616) ◦ Source Harvest: Added
parent_id
for all streams which have parent stream (#15221) ◦ Source Zendesk Chat: is now in Beta (#15959) ◦ Source File is now in Beta (#16001) ◦ New Destination: SQLite (#15018) ◦ New Source: AppFollow Service (#14418) ◦ New Source: Freshcaller Contact Center (#14759) ◦ New Source: Hubplanner Resource Management (#15521) ◦ New Source: LinkedIn Pages (#13098) • Coming soon to Beta: ▪︎ Twilio ▪︎ Slack ▪︎ Chargebee ▪︎ Marketo ▪︎ Paypal TransactionYoutube Analytics • New Features for Existing Connectors ◦ Source Paypal Transactions: Removed OAuth option (#15741) ◦ Source Amazon Seller Partner: add restock inventory reports (#15063) ◦ Source Amazon Ads: Skip API error “Report date is too far in the past.” (#15921) ◦ Source Marketo: 90% unit test coverage (#15817) ◦ Destination BigQuery: Enabling Application Default Credentials (#14784) ◦ Destination Postgres: Add SSL certificates and update normalization (#14743) ◦ Destiantion Cassandra: Replace batch insert with async method (#15894) ◦ Destination Scylla: Handle per-stream state (#15399) ◦ Destination Snowflake: support key pair authentication (#14388) ◦ Destination S3: add LZO compression support (#15394) • Source Bing Ads adding missing columns: (#14862) ◦ account_performance_report, ad_group_performance_report, ad_performance_report, campaign_performance_report, keyward_performance_report ◦ Source File: Convert ‘nan’ to ‘null’ (#15768) ◦ Source File - add support for custom encoding (#15293) ◦ Source Github: PullRequestCommentReactions - re-implemented using GraphQL (#14795) ◦ Source Hubspot: implement new stream to read associations in incremental mode (#15099) ◦ Source Intercom: Update Intercom API to version 2.5 (#15681) ◦ Source Iterable**:** API key is passed in headers (#15670) ◦ Source Jira: Added updates to include issue components and fixes to README files (#15135) ◦ Source MySql: Added SSL certificates to MySql Source (#15044) ◦ Source Okta: return deprovisioned users (#15001) ◦ Source Okta: added parameter ‘start_date’ (#15050) ◦ Source Okta: add resource-sets (incremental supported) (#14700) ◦ Source Oracle: Use Service Name to connect to database (#14953) ◦ Source S3: Support JSON format (#14213) ◦ Source Snowflake: add scope for role for oauth (#15797) and add ability to support different roles during oauth (#15654) ◦ Source Stripe: external account streams (#14357) ◦ Source Typeform: update Forms schema (#15435) • New Features in Airbyte Core ◦ Base Normalization: quote schema name to allow reserved keywords (#14683) ◦ Observability: Export temporal metrics to datadog (#14842) and constant Tags (#15213) and record schema validation errors (#13393) ◦ Airbyte Cloud: Sync Log Summary Doc (#15181) ◦ Self Hosting: Helm Chart updates (#14992 and #15199 and #15374 and #15719) ◦ Webapp: Enable OAuth login (#15414) ◦ Apply shutdown hook to airbyte server to allow graceful termination (#15934) • Ongoing work on our new Low Code CDK ◦ Add request options and state to stream slicers (#14552) ◦ Disable parse-time interpolation in favor of runtime-only (#14923) ◦ handle single records (#15346) ◦ fix so we don’t display yaml when debug flag is turned off (#15383) ◦ fix bug where headers were not passed to cursor interpolation (#15347) ◦ fix parse and format methods (#15326) ◦ read configs from package_data (#15810)
🚨 Breaking changes Reminder that in order to use the new GA Postgres connector, there are a few requirements: • Upgrade your Airbyte platform to
v0.40.0-alpha
or newer • Upgrade your Postgres source connector to
1.0.0
or newer If you are using any of the following destinations, ensure they are also upgraded to the following versions or newer: • destination-azure-blob-storage: 0.1.6 • destination-bigquery: 1.1.14 • destination-cassandra: 0.1.3 • destination-dynamodb: 0.1.5 • destination-kafka: 0.1.10 • destination-keen: 0.2.4 • destination-kinesis: 0.1.4 • destination-mongodb: 0.1.6 • destination-mqtt: (TBD - will be updated) • destination-pubsub: 0.1.6 • destination-pulsar: 0.1.3 • destination-rockset: 0.1.4 • destination-scylla: 0.1.3 🐛 Bug fixes • Destination azure: bump version (#15412) • Destination S3: connector check ignores the s3_bucket_path (#15207) • Source Zendesk Chat: fix CI problem + corrected public docs (#15879) • Source file: fix csv schema discovery (#15870) • Source Amazon Ads: move STATE inside report stream class (#15816) • Show Datatypes Correctly (#15558) • Source Asana: Add cash for project stream (#15749) • Salesforce source: exclude knowledge-article from bulk api (#15901) • Source Amazon Ads: set default value for ‘region’ if needed (#15822) • Source Amazon Ads: define primary_key for all report streams (#15469) • Destination Bigquery: fix changelog and publish new version (#15409) • Destination MongoDB: use SHA256 instead of MD5 (#14561) • Destination Rockset to handle per-stream state (#15395) • Destination Azure Blob Storage: Handle per-stream state (#15318) • Source Facebook Marketing: Updated date validation process (#15327) • Source Freshdesk: fix spec for backward compatibility (#15378) • Source File: bump setup.py versions (#15269) • Source Github: bugfix schemas for streams
deployments
,
workflow_runs
,
teams
(#15049) • Source Github: validate input organizations and repositories (#15730) • Source Google Ads: Fix wrong schema for
ad_group_criterion.topic.path
and shifted
Campaigns
stream to non-managers stream list (#15084) • Source Google Sheets: exposes row batch size config (#15107) • Source Harvest: “started_time” being incorrectly cast as datetime field (#15312) • Source Hubspot: fix 401 for associations (#15156) • Source Harvest: increase unit test coverage at least 90% (#15220) • Source Instagram: ask for pages_show_list,pages_read_engagement scopes during oauth flow (#15802) • Source-MSSQL : special character support in dbname #14824 #15186 (#15268) • Source MSSQL: special character support in dbname for CDC method (#15430) • Source Okta: fix for failed stream on
Json Validation
NPE (#15179) • Source Okta: add permission stream under a custom role (#14739) • Source PayPal Transactions: Updated docs (#15105) • Source PayPal Transactions: increase unit tests (#15098) • Source Postgres: fix bug in intermediate state emission (#15496) • Source SalesForce: changed
DEFAULT_WAIT_TIMEOUT_SECONDS
to 24-hour limit (#15444) • Source Shortio: Changing links primary key (#15066) • Source Snowflake: int columns should be discovered as ints (#15314) • Source Stripe: implement slicing (#15292) • Source Zendesk Support: add
Subscription Plan
check for available streams (#15233)