Hello all! We wanted to give everyone a heads up that we have just published a release of the low-code framework in

airbyte-cdk:0.29.0

. This release contains a number of breaking changes that are intended to improve the overall usability of the language by reorganizing certain concepts, renaming, reducing some field duplication, and removal of fields that are seldom used. The Airbyte team has already worked to migrate all existing connectors in the Airbyte repository to support the changes to the language. However, connectors that are currently in-progress or that have not been merged as of Wednesday 2/15 will have to be adjusted to accommodate the new changes. Here are a list of the major changes to components that might affect your manifests: • We have renamed

$options

to

$parameters

• We have changed the notation for referencing other components to the JSON schema notation (

$ref: "#/definitions/requester"

) • We have deprecated the SimpleRetriever’s

stream_slicer

in favor of two individual concepts. • Stream’s will define an

incremental_sync

field which is responsible for defining how we would support incremental syncs using a cursor field.

DatetimeStreamSlicer

has been renamed to

DatetimeBasedCursor

and can be used for this field. • Retriever’s will now define a

partition_router

field. We have renamed the remaining slicers to be called

SubstreamPartitionRouter

and

ListPartitionRouter

, both of which can be used here as they already have been. • We’ve deprecated the CartesianProductStreamSlicer because

partition_router

can accept a list of values and will generate that same cartesian product • DefaultPaginator no longer has a

url_base

field. Moving forward, paginators will derive the

url_base

from the HttpRequester. There are some unique cases for connectors that implement a custom Retriever, which I can go into more detail if necessary. •

primary_key

and

name

no longer need to be defined on Retrievers or Requesters. They will be derived from the stream’s definition • Stream’s no longer define a

stream_cursor_field

and will derive it from the

incremental_sync

.

checkpoint_interval

has also been deprecated • DpathExtractor

field_pointer

has been renamed

field_path

• We no longer allow for using

RequestOption

with

inject_into

set to path. We now have a dedicated

RequestPath

component moving forward. In addition, a change was released a couple of weeks ago where we are now validating manifests against the declarative_component_schema.yaml which is a handwritten schema maintained by our team. You can use that file as a reference document as you write or update manifests. Thank you being contributors and early access users of the low-code framework and please feel free to reach out to the Airbyte team if you run into issues migrating your in flight connectors and we will try to help ease the process or point you towards sample connectors for your use case.

Thanks for the heads up, was wondering why one of our custom connectors was failing to be updated. Is there a non-Slack announcement for these changes as well, that we can link to share (for those on our team who are not on this channel)

Hi Gergely, we don’t currently have a formal means for communicating changes outside of slack at the moment. This is something we are still thinking through but we want to put a plan in place for communicating improvements or changes more broadly as the language reaches maturity and moves to General Availability. Thanks for the feedback!

FWIW we’re going to publicize our changelog soon. This will allow you to point team members towards such announcements.

If you need more information, I think we did a good job at documenting these changes here : https://github.com/airbytehq/airbyte/issues/21926 This is a specific issue to update one of our projects using the airbyte-cdk and it contains all the breaking changes with reference to the issues and PRs

anyone else managed to suscesfully build a new custom connector using the connector builder and generator since these changes?

Hey @Josh Jeffries! Without more information, my best guess is that the connector builder has generated a version of the manifest that is using compatible with

airbyte-cdk==0.28.1

but not

airbyte-cdk==0.29.0

which your pip install might be resolving. Can you provide more information as to the issue you are encountering? The connector builder should be updated soon to create manifest that takes into account those breaking changes. We’ll keep you posted!

so when running the python main.py spec after the requirements i get errors on it (see 1st pic). Some have suggested to alter the setup.py and include

airbyte-cdk==0.28.1

(unless i am looking at the wrong setup.py mine is actually showing up as

airbyte-cdk~=0.1

. if i do change it, to 28.1 and run the pip install again, it gets a little further but still has errors (see pic 2)

It seems like the manifest you have provided is not interpreted as a YAML object but as a string. Can you share the manifest that was generated by the connector builder with me and I’ll try running it on my side? If you don’t want it to be public, you can share it by DM

Hey Maxime, this is actually pre uploading the YAML from the connector build, this is right after doing the ./generate.sh and setting the venv ie

cd ../../connectors/source-jobadder
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python main.py spec

That a good point! We haven’t updated the generator. Therefore, I’ll update this shortly. Thanks for letting us know! Knowing this, my expectation is that if you update

setup.py

to set

airbyte-cdk==0.28.1

before running

pip install -r requirements.txt

, the manifest generated by the Connector Builder should work there

👍 ill give it a go

Let me know if there is anything and thanks for the feedback

@Maxime Carbonneau-Leclerc (Airbyte) Yep you were right, changing the setup.py and using the yaml from the connector build does work (well kind of i am getting a Fetching cpnnector failed error, but pretty sure me lol) Cheers for the help

Anytime! We will work on improving how we do breaking changes. Connector Builder is new and we’re still ironing out the details of how we update it to be aligned with the airbyte-cdk