Hey Airbyte team, how to better do API development? As i test i added a log statement in one of API method. In order to see the log, i have to stop docker, build again with

SUB_BUILD=PLATFORM ./gradlew build

and then make docker up again. This is such a productivity killer. i must be missing something. I already try to build only server

./gradlew :airbyte-server:build

also tried

--continuous

flag so gradle keep building. But i must stop docker and start again to see it in effect. i guess i understand why docker needs to be restart so it can pick up new jar. but how to do API development in more easy/better way. Can someone please assist? I am trying to debug and create new API/methods. Thank you very much for your help in advance.

Is there any way to set default value for stream? I'm receiving

None

where I need empty list

[]

Hey team, I have been working with @Francesco F on an Aircall Connector via the Connector builder UI, and he has gotten stuck loading it into the platform. Can you provide some thoughts on what else to check here? @Francesco F can you upload a copy of your YAML?

hi

i was trying to build a custom connector and got the below error when running the generate.sh . please help

Hi team, is there any docs about writing a strict-encrypt connector in python?

Hey Team, I have created the custom connector for HTTP-API and trying to create a new connection using Airbyte UI. I have completed the first step by adding a newly created connector image to Airbyte source. But when I try to add destination for the source I am getting an unknown error which is I am not aware of, I have attached the screenshot for the error.

I am trying to create a new connector using the UI, but the documentation on what to do with the exported yaml seems to be either incorrect or outdated or I am just confused. I am following Connector Builder UI | Airbyte Documentation 1. I start to have issues in the “Exporting the YAML” section, when it talks about overwrite the contents. It references a yaml file with the name of the connector but I don’t get a file like that I get a manifest.yaml. I used the connector generator talked about in this doc: Step 1: Generate the source connector project locally | Airbyte Documentation. I am able to follow step 1 and 2 without issues but I get lost in step 3 when it talks about a yaml file with same name as the connector, again I just have a manifest.yaml where the named file should be. If I take the content of the yaml from the UI builder and put it in manifest.yaml I get a successful connection when I run main.py check. In step 4 it talks about creating schema files, but doesn’t reference how to use the schema built in the UI builder. Is there any documentation for the UI Builder connector process, or anything I am missing?

🙏

Hello all! I've been kicking the tires with low-code connector development attempting to build a source connector for a custom API. I could get through most of the tutorial but ran into two issues on the final step (adding my connector to Airbyte and setting up a connection with it). 1. I couldn't add the connector by adding to

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

as documented here. I removed all local containers and volumes and started fresh with

docker compose up

but my connector doesn't appear under "Settings > Sources". I tried running

./gradlew :airbyte-config:init:processResources

to add it manually but it fails, presumedly because it's looking for my custom connector Docker image in a GCP-hosted repository instead of my local images (see attached image). Is this expected/configurable? 2. I was able to install via the UI, but hit another snag after setting up the source when it attempted to fetch the source schema (see second screenshot). Running Airbyte in dev mode (as I mentioned here) resolved this, but I wouldn't think it would be necessary. Thanks!

Hello all! We wanted to give everyone a heads up that we have just published a release of the low-code framework in

airbyte-cdk:0.29.0

. This release contains a number of breaking changes that are intended to improve the overall usability of the language by reorganizing certain concepts, renaming, reducing some field duplication, and removal of fields that are seldom used. The Airbyte team has already worked to migrate all existing connectors in the Airbyte repository to support the changes to the language. However, connectors that are currently in-progress or that have not been merged as of Wednesday 2/15 will have to be adjusted to accommodate the new changes. Here are a list of the major changes to components that might affect your manifests: • We have renamed

$options

to

$parameters

• We have changed the notation for referencing other components to the JSON schema notation (

$ref: "#/definitions/requester"

) • We have deprecated the SimpleRetriever’s

stream_slicer

in favor of two individual concepts. • Stream’s will define an

incremental_sync

field which is responsible for defining how we would support incremental syncs using a cursor field.

DatetimeStreamSlicer

has been renamed to

DatetimeBasedCursor

and can be used for this field. • Retriever’s will now define a

partition_router

field. We have renamed the remaining slicers to be called

SubstreamPartitionRouter

and

ListPartitionRouter

, both of which can be used here as they already have been. • We’ve deprecated the CartesianProductStreamSlicer because

partition_router

can accept a list of values and will generate that same cartesian product • DefaultPaginator no longer has a

url_base

field. Moving forward, paginators will derive the

url_base

from the HttpRequester. There are some unique cases for connectors that implement a custom Retriever, which I can go into more detail if necessary. •

primary_key

and

name

no longer need to be defined on Retrievers or Requesters. They will be derived from the stream’s definition • Stream’s no longer define a

stream_cursor_field

and will derive it from the

incremental_sync

.

checkpoint_interval

has also been deprecated • DpathExtractor

field_pointer

has been renamed

field_path

• We no longer allow for using

RequestOption

with

inject_into

set to path. We now have a dedicated

RequestPath

component moving forward. In addition, a change was released a couple of weeks ago where we are now validating manifests against the declarative_component_schema.yaml which is a handwritten schema maintained by our team. You can use that file as a reference document as you write or update manifests. Thank you being contributors and early access users of the low-code framework and please feel free to reach out to the Airbyte team if you run into issues migrating your in flight connectors and we will try to help ease the process or point you towards sample connectors for your use case.

Hey, im working on the amazon seller partner connector atm, trying to fix the problem of the access token not getting refreshed before expiring. But none of my changes or breakpoints outside of my connector are getting used while testing Which sucks, because i think that there is a bug in the authenticator since the token should be refreshed according to the code. I use pdb as debugger and start my Test with: python -m pdb main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json Working with VS Code and PyCharm Did i miss something? Do i need to add the cdk in my command to start the test?

👋 as of airbyte

v0.40.32

we noticed that all keys in the airbyte record data passed to our destination now comes in as strings. For example for

zendesk_support schedules

, pre

v0.40.32

we see schedules come from

record.data

as (notice that id, start and end time are all ints):

{'id': 1260800006989, 'name': 'M-F 9-5pm PT', 'time_zone': 'US', 'created_at': '2021-04-22T00:32:27Z', 'updated_at': '2022-12-31T01:06:17Z', 'intervals': [{'start_time': 1800, 'end_time': 2460}, {'start_time': 3240, 'end_time': 3900}, ...]}

now they come in as strings:

{'id': '1260800006989', 'name': 'M-F 9-5pm PT', 'time_zone': 'US', 'created_at': '2021-04-22T00:32:27Z', 'updated_at': '2022-12-31T01:06:17Z', 'intervals': [{'start_time': '1800', 'end_time': '2460'}, {'start_time': '3240', 'end_time': '3900'}, ...]}

It is causing issues in our destination. Is that the expected behavior now? Thanks for the help 🙏

Hey team. We use Airbyte to sync between Airtable and a data warehouse. The Airtable schemas change quite a lot, and it's pretty annoying to have to keep refreshing the source schema and resetting the data. Therefore, I'd like to programmatically trigger an update of refresh and reset of my airtable source/connection on a nightly basis. I previously opened a similar ticket. The attached script show my attempt so far, but it doesn't work because the

write_discover_catalog_result

endpoint requires a

configurationHash

that I don't know how to create. (The docs for `write_discover_catalog_result` also say "Should only called from worker", so I assume I'm doing something wrong). Can anybody give me some pointers how to either generate the source config hash, or if there is a better way to update the source definition? (ps. I know this is not strictly a CDK question, and more about the usage of the API, but I hoped I might get a better answer from this channel 🙏)

Hey Everyone, i am currently bulding a custom connector that i want to use the incremental sync feature, so far everything is looking good, apart from trying to add in a Start time request option. My data source (JobAdder a recruitment CRM - api doc https://api.jobadder.com/v2/docs#tag/Note-API:Notes) supports a request parameter of updatedAt but requires a > to be prefixed to the date, any idea how i can do that?

Afternoon again, I am not sure if i have found a bug or not on the cursor pagination on the connector builder. for every new page, it seems to be duplicating the request parameters if you have specified. For instance this is the request for the first page

{
  "url": "<https://api.jobadder.com/v2/users>",
  "parameters": {
    "updatedAt": [
      ">2000-01-05T00:00:00Z"
    ],
    "include": [
      "inactive, deleted"
    ]
  },
  "body": null,
  "headers": {
    "User-Agent": "python-requests/2.28.2",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer 86f6f5d182afa48305d9ea85a439d1b9"
  },
  "http_method": "GET"
}

the updatedAt and the include are specified request parameters, on the next page the request looks like

{
  "url": "<https://eu2api.jobadder.com/v2/users>",
  "parameters": {
    "updatedAt": [
      ">2000-01-05T00:00:00Z",
      ">2000-01-05T00:00:00Z"
    ],
    "include": [
      "Inactive",
      "Deleted",
      "inactive, deleted"
    ],
    "offset": [
      "100"
    ]
  },
  "body": null,
  "headers": {
    "User-Agent": "python-requests/2.28.2",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer 86f6f5d182afa48305d9ea85a439d1b9"
  },
  "http_method": "GET"
}

and again on the next one its

{
  "url": "<https://eu2api.jobadder.com/v2/users>",
  "parameters": {
    "updatedAt": [
      ">2000-01-05T00:00:00Z",
      ">2000-01-05T00:00:00Z",
      ">2000-01-05T00:00:00Z"
    ],
    "include": [
      "Inactive",
      "Deleted",
      "Inactive",
      "Deleted",
      "inactive, deleted"
    ],
    "offset": [
      "200"
    ]
  },
  "body": null,
  "headers": {
    "User-Agent": "python-requests/2.28.2",
    "Accept-Encoding": "gzip, deflate",
    "Accept": "*/*",
    "Connection": "keep-alive",
    "Authorization": "Bearer 86f6f5d182afa48305d9ea85a439d1b9"
  },
  "http_method": "GET"
}

also related it doesn't seem to be passing through the page_size either

👋 Where in the CDK is the

airbyte_emitted_at

set for every record ?

Hey guys, any reason in particular "top 1" is used on incremental syncs? Me, our data engineer and DBA have been scratching our heads for a while, and we assume it may be part of airbyte's governance for new connectors. If this isn't a fixed airbyte thing, is there a way to build more perfomanct connectors in the webapp?

Hello! I wanted to ask for guidance on implementing a connector using the CDK. I want to create a TikTok connector that pulls organic user (followers) and video data (likes, shares, views) using TikTok API v2. I can't use the TikTok Marketing connector since it provides different data (Ads, Adgroups, etc.), not the organic data we need. I started making the connector, but ran into issues on authentication: TikTok currently only supports OAuth2 for authentication. But I think the CDK only supports OAuth2 when you already have a refresh token, right, and not the full cycle? Additionally, you have to sign up on TikTok'd developer platform and register your app and domain, and they'll only allow redirects after authorization to the domain you registered -- which I think would be hard if we're self-hosting Airbyte on our own machines. (Though the client we're developing this for is using Airbyte Cloud -- so I guess we could register the Airbyte Cloud domain? But then I don't know how connector development goes for Airbyte Cloud) I also checked out the Research API they have available, but since this is for a business, I don't qualify for access under their requirements. Would any of you have advice on how to go about this and be able to implement the OAuth2 authentication? Right now, because of the domain registration requirement AIrbyte has, the only way I can see feasibly having a connector is if Airbyte itself deployed the connector (so the Airbyte Cloud domain is registered under Tiktok), and even then it would only be usable with Airbyte Cloud. 😢 I'd really appreciate any insights on this, I'm really not sure how to approach this particular issue. Thanks!

Hi! I am using the CDK to develop a custom HTTP source connector where in order to collect the data from the source, the connector should send a POST request with two request body fields that change for any new datapoint. My question is how/where should I pass these request body fields to the source connector?

Currently we are playing with the idea of programming a Google My Business integration in-house, which gives us the locations metrics. Is there anyone who is currently already on it or is in the planning at Airbyte?

Hello here 👋 I opened a PR to fix an issue related to a marketing lookback window on

Bing Ads

Source. Would anyone here be willing to review the implementation (I used Python CDK)? 🙂

Hi all, I want to test if my changes in one of the sources is working correctly, before opening a PR. But I can’t use the built in unit test running in IntelliJ to run the python unit tests. The tests are run when I execute

gradle test

, but that option is not ideal, as I don’t wish to escape my IDE 😄

I created a python virtual env sdk entry poinint to the .venv of the connector I am working on.

I pushed a PR with a bugfix in the datasource source connector. However, my git configuration did not have the correct e-mail address configured, so the CLA assistent is not recognizing 1 of the 2 committers. Should I apply the fix on a fresh checkout? Or can this be fixed on your end

Is there a good example of how to implement oauth2 authentication using the cdk?

hey y’all! I’m trying to add a new connector and I’m following these docs but the connector doesn’t show up in under destinations in the platform UI. I see in logs that the local catalog is found, but every time I refresh the destinations page it downloads from the RemoteDefinitionsProvider. Any tips?

Hello, I’m attempting to create an incremental build using grapqhql. I’ve been able to return data by inserting the query into

request_body_data

. This is great for a full refresh, but again, I’d like to build an incremental build. Does anyone know of a dynamic way to insert data into

response_body_data

? In the image attached, I show the configuration that works. I’ve attempted to execute the line before last (commented out) which has a dynamic variable, but I can’t get the timestamp to render. Instead, I see no data passed on to the request whereas I can see the entire query when using line 81.

Is there documentation for how change data capture is represented inside airbyte protocol? I can’t find it after searching everywhere

is it possible to generate streams schema from openapi