user
08/09/2024, 4:58 PMuser
08/15/2024, 4:26 PMcursor_field
for TicketMetrics
stream to be generated_timestamp
(#43446)
34c76fb ✨ Source Okta: Authentication with private key (OAuth 2.0) support (#43382)
a7a9688 refactor(source-mixpanel): migrate to CDK v3 (#41969)
42f7d56 Source workable: Migrate to manifest-only (#44044)
f3ca2b4 Source whisky-hunter: Migrate to manifest-only (#44045)
b73f80e Source waiteraid: Migrate to manifest-only (#44047)
4565c96 Source vitally: Migrate to manifest-only (#44049)
782ce3f Source aha: Migrate to manifest-only (#44042)
0cfdf7e Revert "Temporarily disable required reviews on Community manifest-only connectors" (#44050)
cba3ccd Destination S3: Deferred deletes on sync success in OVERWRITE (#42579)
325525d refactor(source-sftp-bulk) bump CDK version to v4 (#44028)
d4322c7 Source weatherstack: Migrate to manifest-only (#44046)
9330801 Temporarily disable required reviews on Community manifest-only connectors (#44004)
34f58fa Destination databricks: Remove user/pass auth (#40712)
adc5f19 metadata-service [orchestrator]: fix typo preventing correct stal…
airbytehq/airbyteuser
08/15/2024, 8:30 PMuser
08/17/2024, 1:12 AMuser
08/20/2024, 6:17 PMtime_tracking
, time_tracking_tags
, team_goals
, space_tags
, team_custom_fields
, list_custom_fields
, list_comments
, Add error handlers, Parent stream flow correction, Update spec (#44180)
6b60095 Source Monday: Add Boards name to items
stream (#44201)
6525eb4 .github: update CODEOWNERS with Bulk CDK paths (#44405)
8813e4c 🤖 Cut version 4.5.2 of source-declarative-manifest
3da00a8 🤖 patch bump Python CDK to version 4.5.2
45d5198 [airbyte-cdk] refactor error handling in abstract source (#40571)
c67c5ba airbyte-ci: generate SBOM on publish (#44377)
5023ac9 source-basecamp contribution from natikgadzhi (#43952)
9d3bcd8 🐛 Source Hubspot : Fix engagements streams associations configurations (#43381)
f05de59 [source-microsoft-sharepoint] - Migrate to CDK v4.3.0 (#42983)
361dc42 Bump aiohttp from 3.9.5 to 3.10.2 in /airbyte-ci/connectors/live-tests (<https://github.com/airbytehq/airbyte/pull/43709…
airbytehq/airbyteuser
08/27/2024, 8:19 PMuser
08/27/2024, 8:47 PMuser
08/30/2024, 9:28 PMorjson
to speed up parsing (#44829)
4a9468a chore(source-gcs): check AirbyteTracedException raised unit tests for file-based sources (#44770)
d141d18 Destinations - Increase Jackson parse limit to 100MB (#44476)
5478a4b fix(source-gcs): fix empty file list when prefix is empty (#44796)
18a8e5f MSSQL and debezium improvements (#44759)
7debf35 Bulk CDK alerts go to the right slack channel (#44867)
abd9da9 bulk-cdk: source output performance improvements (#44865)
643c800 🤖 Cut version 4.6.1 of source-declarative-manifest
75f9594 🤖 patch bump Python CDK to version 4.6.1
1060161 [airbyte-cdk] - Update JSON Error Message Parser to return additional error message fields or default to dict (#44010)
af98a81 [certified db sources] adopt latest CDK to use error translation bug fixes (#44841)
a5b28f4 🤖 Cut version 4.6.0 of source-declarative-manifest
a6528e7 🤖 minor bump Python CDK to version 4.6.0
fc8cd5a fix(python-cdk): add user friendly message for encoding errors (#44438)
dc6a1cc docs(source-instagram): fix docs to match the template (#44856)
6cbba8d [source-mongodb] use escaped dot in wass streamlist (#44846)
64fdb80 live-test: improve error message on connection retrieval (#44809)
708a9a1 cdk connectors test: fix regression test run (#44849)
bcfe72b test(source-google-ads): fix abnormal state for ad_group_criterion
and campaign_criterion
(#44816)
4d54106 docs(source-bing-ads): fix docs to match the template (#44817)
a32e94e community-ci: support regression tests on forks (#44813)
<https://github.com/airb…
airbytehq/airbyteJack Weatherilt
09/02/2024, 8:58 PMuser
09/04/2024, 11:55 PMabctl version
command in oss-quickstart (#44927)
b1b1187 metadata-service[orchestrator]: fix broken test (#45120)
84f3b03 fix(source-github): fix state handling for stream WorkflowRuns (#44592)
1f2fd6e 🐛 Source Shopify: Have message and description be nullable for custom_collections delet… (#45116)
af58faa Unit tests for streams manager (#45090)
5bf11d9 Destination Databricks - clarify catalog name option (#45115)
427e8fd feat(airbyte-ci): include components.py in manifest-only build (#44879)
095d30e State manager test, rational range behavior throughout (#44945)
2073513 docs: remove lingering mentions of dbt core (#45110)
be6b5ef bulk-cdk-toolkits-extract-jdbc: noop JdbcCheckQueries when no queries defined (#44842)
f85baeb (source-s3) - Upgrade CDK 3.9.6 (#44908)
09fef1c Fix pagination issue while setting is_resumable properly (#45101)
5d1c2bf doc: app-name is now called appName (#44882)
f805842 feat(source-hardcoded-records): bump CDK version (#45097)
b4ccb27 🤖 Cut version 5.0.1 of source-declarative-manifest
844f52b 🤖 patch bump Python CDK to version 5.0.1
fa6aa2b fix(source-declarative-manifest): fix models (#45088)
6eb0d36 🤖 Cut version 5.0.0 of source-declarative-manifest
b1df9b8 🤖 major bump Python CDK to version 5.0.0
df34893 feat(airbyte-cdk): replace pydantic BaseModel
with dataclasses
+ serpyco-rs
in protocol (#44444)
21fddbd re-enable dagger cloud token in airbyte-ci (#45080)
c0e629f source-savvycal contribution from natikgadzhi (#45069)
34eb16d source-calendly contribution from natikgadzhi (#45067)
477689b 🐙 source-bing-ads: run up-to-date pipeline [2024-08-31] (#44276)
8afedec 🐙 source-google-ads: run up-to-date pipeline [2024-08-31] (#44326)
374e805 🐙 destination-qdrant: run up-to-date pipeline [2024-08-31] (#44678)
b593c07 🐙 source-yandex-metrica: run up-to-date pipeline [2024-08-31] (#44702)
f297a4b 🐙 source-greenhouse: run up-to-date pipeline [2024-08-31] (#44755)
5cf3b9a 🐙 source-public-apis: run up-to-date pipeline [2024-08-31] (#44947)
7d8d11c 🐙 source-my-hours: run up-to-date pipeline [2024-08-31] (#44948)
de474f0 🐙 source-gridly: run up-to-date pipeline [2024-08-31] (#44949)
320ed8e 🐙 source-linkedin-ads: run up-to-date pipeline [2024-08-31] (#44950)
d44ab19 🐙 source-tplcentral: run up-to-date pipeline [2024-08-31] (#44951)
f50bc69 🐙 destination-duckdb: run up-to-date pipel…
airbytehq/airbyteuser
09/06/2024, 6:23 PM/calls/extensive
(#45117)
2d94c53 🤖 Cut version 5.2.1 of source-declarative-manifest
d7f85ed 🤖 patch bump Python CDK to version 5.2.1
caa12ab docs(airbyte-cdk): Fix error in incremental sync docs (#45194)
950ea06 🤖 Cut version 5.2.0 of source-declarative-manifest
9700e4b 🤖 minor bump Python CDK to version 5.2.0
2fa35ab feat(airbyte-cdk): Add Global Parent State Cursor (#39593)
04f40f9 metadata-service[orchestrator]: fix stale metadata detection job (#45188)
337e1fc feat(source-zendesk-support): update CDK to v5 (#45187)
b5756b7 🤖 Cut version 5.1.0 of source-declarative-manifest
e73a4dc 🤖 minor bump Python CDK to version 5.1.0
03b7e1a feat(airbyte-cdk): Add limitation for number of partitions to PerPartitionCursor (#42406)
1de50aa [source-mysqlv2] A new mysqlv2 (#44606)
081a0ca Bulk Load CDK: Unit tests for memory manager (#45091)
6730a3b destination-s3: don't reuse names of existing objects (#45143)
e6d6b8e Source Looker: Enable latest version in Cloud registry (#45161)
f1812e3 (source-tiktok-marketing) - Upgrade CDK 4.6.1 (#44910)
011f40b Revert "fix: allow review requirements check on forks (#44404)" (#45175)
155d58d docs(airbyte-cdk): update migration guide (#45153)
dd2d443 gradle: remove deps.toml (#45160)
3b79bd7 metadata-service[orchestrator]: fix missing isReleaseCandidate field (#45154)
458e96e [source-mssql] update datetimeoffset format (#45142)
airbytehq/airbyteuser
09/13/2024, 6:06 PMsuccessful
(#45426)
647a3e8 Docs: add note about migrating docker uses external integrations (#45406)
5dbce81 airbyte-ci: improve QA checks for airbyte-enterprise (#45393)
a135236 add .kotlin to the .gitignore (#45404)
19335d6 [source-us-census] fix empty fields after sync (#45331)
7056428 Bulk Load CDK: State -> Checkpoint & flush at end (#45377)
b00dac8 feat(source-stripe): ref stream Payment Methods
(#44891)
e679191 ✨ Source Sendgrid: Move contacts stream to async declarative component (#45191)
e4fec50 bulk-cdk: improve AirbyteConnectorRunner and CliRunner (#45374)
03584d5 [source-mysql/mssql] Fix state manager on determining non-resumable streams (#45181)
b6825ee Bulk Load CDK: Simply Interface & Add Check (#45369)
41a242b Remove logging of unhandled debezium events (#45368)
e4c26e8 live-test: improve error message when the selected connection did not run the control version (#45335)
8d7b191 [Source-mysql-v2] : Support to add CDC option (#45366)
5761f46 🤖 Cut version 5.5.2 of source-declarative-manifest
9781dd5 🤖 patch bump Python CDK to version 5.5.2
e6659ad airbyte-cdk: poetry lock before release (#45365)
b672bc0 Remove pandas from extras in file-based (#45362)
e00eaf6 [Source-mysql-v2] : Fix tests to be passing on local (#45363)
eb018e6 🤖 Cut version 5.5.1 of source-declarative-manifest
40a7c94 🤖 patch bump Python CDK to version 5.5.1
9bcfa3d source-mysql-v2: set supportLevel to community (#45341)
b2e63f2 airbyte-ci: relax docs qa checks when sl == 0 (#45360)
c3c8423 .github: fix airbyte-ci-release.yml (#45361)
21348af ref(source-stripe): remove stripe py-package (#45348)
ba63c03 feat(source-zendesk-support): add automations and triggers streams (#44610)
8332ddc 🤖 Cut version 5.5.0 of source-declarative-manifest
c3048c6 🤖 minor bump Python CDK to version 5.5.0
6baf254 feat(cdk): add async job components (#45178)
254f34a 🤖 Cut version 5.4.0 of source-declarative-manifest
698be7e 🤖 minor bump Python CDK to version 5.4.0
e3ce82e feat(airbyte-cdk): add global_state => per_partition transformation (#45122)
bd20b74 🤖 Cut version 5.3.0 of source-declarative-manifest
e2d8c23 🤖 minor bump Python CDK to version 5.3.0
41b8585 fix(connector-builder): add flag to disable cache (#45095)
52744ec feat(source-getlago): ignore 405 response for customer usage (#45193)
bb1833c Destination Databricks: Create namespace if missing in CHECK (#45208)
58989b0 metadata-service[orchestrator]: avoid "too large" errors in generate_registry_reports (#45336)
7ffcd3d feat(source-shopify): update CDK to v5 (#45190)
f216a60 📚 docs: Enable ERDs for 10 top sources. (<https://github.com/airbytehq/airb…
airbytehq/airbyteEmmanuel Lecomte
09/16/2024, 2:12 PMuser
09/17/2024, 5:59 PMuser
09/17/2024, 8:24 PMuser
09/18/2024, 2:26 PMuser
09/24/2024, 1:04 PM_airbyte_meta.errrors
for large records or mistyped data. Read more in our </airbytehq/airbyte/blob/v1.0.0/using-airbyte/core-concepts/typing-deduping#_airbyte_meta-errors|typing & deduping documentation>.
• Large initial CDC syncs are gracefully handled by consuming WAL and acknowledges logs periodically to ensure disk space is freed in a timely manner.
• </airbytehq/airbyte/blob/v1.0.0/understanding-airbyte/jobs#workloads|Workloads> provide a more scalable and reliable architecture to run sync jobs by separating scheduling and orchestration from data movement tasks. This improvement unlocks more automated management of workloads by managing job spikes and enables horizonal scaling (for Cloud and Enterprise users).
Tackling the long-tail of connectors
• We launched our connector Marketplace for our community-maintained connectors. Automated testing ensures our connectors stay high quality, and most connectors have been migrated to low/no-code.
• The Connector Builder seamlessly switches betweem UI and YAML modes, supports custom components, and contains extensive testing and error handling.
• AI Assist now builds connectors from scratch for you with just a link to the API docs.
• Anyone can now contribute new connectors directly from the builder to add a new connector directly to Airbyte's Marketplace. Additionally, connectors can be forked directly from the UI for faster edits, ensuring our connectors stay up-to-date.
• Airbyte supports writing to several vector store destinations, including pgVector, Pinecone, and Snowflake Cortex.
Self-Managed Enterprise
Self-Managed Enterprise extends on Airbyte 1.0 by introducing new classes of functionality: multitenancy and access management, enterprise source connectors , sensitive data masking and enterprise support – all while data never leaves your infrastructure.
• Support for Okta and OpenID Connect (OIDC) SSO
• Availability of </airbytehq/airbyte/blob/v1.0.0/access-management/rbac|Role-based Access Control> and multitenancy
• Columns can now be hashed to protect sensitive data
Announcements
• To ensure adherence to security best practices, Airbyte is migrating all connectors to non-root versions. It is highly recommended that you upgrade your platform version to v0.63.9 or later before October 2024 to ensure your syncs continue to succeed.
• As we prepare to deprecate Docker Compose, we published a </airbytehq/airbyte/blob/v1.0.0/using-airbyte/getting-started/oss-quickstart#migrating-from-docker-compose-optional|migration guide> for those migrating from Docker Compose to abctl.
airbytehq/airbyteMatt Newman
09/27/2024, 8:16 PMuser
10/03/2024, 6:26 PMSchema Tab▾
Patrick Nilan
10/14/2024, 4:50 PMuser
11/07/2024, 4:53 PMfile-transfers-sftp-bulk-source▾
<http://my-registry.foo.com|my-registry.foo.com>
. The value provided in the registry
field must have a trailing slash:
global:
image:
registry: my-registry.foo.com/
• XML Support in the Connector Builder: Airbyte now supports custom API connectors built via the Airbyte Connector Builder which return responses as XML format.
xml-builder▾
rbac-1.2▾
[DEPRECATED] Marcos Marx
[DEPRECATED] Marcos Marx
user
12/09/2024, 10:12 PMImproved Log Viewer▾
cluster-admin
user. Your Kubernetes namespace will also require the following annotations to ensure that all of the Airbyte resources run as UID 1000 and GID 1000. If you run into any edge cases or issues deploying to OpenShift, as always, please let us know by opening a GitHub issue.
openshift.io/sa.scc.supplemental-groups: 1000/1
openshift.io/sa.scc.uid-range: 1000/1
🚀 Self-Managed Enterprise Changes
• Compliance Mappers in Airbyte API: Airbyte Self-Managed Enterprise now supports field hashing, field encryption (with self-managed encryption keys), field renaming and row filtering for connections from a common interface available in the Airbyte API. This functionality is coming soon to the Airbyte UI. To get started, see our API documentation.
Mappers API Example▾
Lake (Airbyte)
12/10/2024, 10:48 PMKevin Lefevre
12/17/2024, 6:40 PM1.3.1
? 🙏
I can not see any changelog
thanks 🙂Ashley Tsoi
01/14/2025, 7:48 PMMichael Hult
03/07/2025, 6:43 PMfull_refresh_overwrite_deduped
Wesley Verheul
03/10/2025, 9:58 AMairbyte-platform
. The current version on the main branch is 6.36.3 but in the latest 1.5.0 release of airbyte the CDK version is still on 6.13. Does anyone know when the CDK upgrade will be released?