thanks @Alexandre Girard (Airbyte)

Hi guys! How are yall doing? Let me explain my question: I’m using the UI CDK builder to build a very simple connector, is an API with just one stream but it’s paginated. As you can see in the screen shot I just uploaded below, the nex_page_url is returned in the API response and I just need to keep fetching the data until there’s no more next_page_url value. Can someone help me with this? It may sound very silly doubt but I’m really struggling with that!

I think I have weaviate fixed - ive added manual schema insertion but im hoping i havent fucked anything up too much lol

Stupid q - im trying to put a PR together for my Weaviate destination connector fix to generate schema via the Airbyte Configured Catalog / Stream. Ive forked airbyte and made a branch, but a push to the branch is errororing with:

v0.44.1-helm -> v0.44.1-helm (refusing to allow a Personal Access Token to create or update workflow .github/workflows/build-report.yml

without

workflow

scope)` im assuming im missing a step in my config.

Hi Everyone, im using UI to build Google DCM source connector, when using path url “https://dfareporting.googleapis.com/dfareporting/v4/userprofiles/{id}/creatives” im getting valid response. so the connection is successful but anyone know how to get all the dimensions and metrics?

@viritha vanama is working on building out a low code version of the Google Campaign Manager source. Viritha Can you share the error message you were getting? I think from what I saw it appears to be an Oauth issue. I haven't tested Oauth with our Connector builder, so I'm not sure about the problem exactly so was hoping for more 👀

its not oauth issue, im receiving the correct response

my question is what should be the path url to get all the metrics and dimensions( i only tried to get creatives above)

Hi friends. Ive submitted a PR to fix a bug and improve performance in the weaviate connector here: https://github.com/airbytehq/airbyte/pull/24524

This is my first contribution to Airbyte and im barely familiar with it - apologies for any mis-steps

Hello everyone, Does anyone know of connectors with nested streams implemented in them?

Hello guys, I'm trying to use connector-build to create a new source that returns data from an api. I have some difficulties because in this API, I need to follow some steps before accessing the data, and connector-build doesn't provide an authentication method needed to return the session id. See below the steps I need to follow in order to access the data: • Get access token • Get session id (cookie) • List campaign data using session cookie. With python, I do POST, and authenticate myself by passing the login info through the body. However, in airbyte I'm not getting any example to be able to build my connector. Can someone help me by providing a case similar to this one where I always need to pass a cookie to access certain data?

@Andy Yeo (Airbyte) @Alexandre Girard (Airbyte) @Conor Barber (Airbyte) @Erica Struthers (Airbyte) Do we have any example of low code connector where we have implemented Oauth2 authentication, paginator and nested streams. I have tried everything using documentation but its not working. I am stuck at this point where i can fetch all the data but paginator and nest streams dont work. I have raised many tickets and sent questions on slack as well. Let me know in case of any example if we have used the above request in low code connector.

Hello Guys i started to do CDC from postgres(VM instance in gcp) to Bigquery using airbyte . I have seen 2 documentation for CDC in both documentation they are using Loading methods as GCS staging. Is it necessary to use that for CDC can I use standard inserts?

I’m trying to develop a connector for Unleashed. The latest hurdle I am trying to overcome is that the dates are given as timestamps in the following format “/Date(1600300800000)/”. What is the best way of going about this? The idea of a custom transformation is scary, and I’m not even sure how I would go about extracting the timestamp from this string and converting it in SQL. Any help would be greatly appreciated! Thank you!

I'm developing a source connector for Fortnox, and are using an OAuth Authentication Code Grant type for authentication. I need to supply ClientId ClientSecret and RefreshToken to configure this connector. My problem is the following: Whenever the refresh token is used to get a new access token a new refresh token is issued and the old one is invalidated, (see https://datatracker.ietf.org/doc/html/draft-ietf-oauth-security-topics#section-4.14.2 under Refresh Token Rotation) The question I have is, if there is any possibility of storing state in the source connector, or somehow update the config to always store the most current refreshtoken in use. I'm thinking that storing the current refresh token on the connection between source and destination is not a good solution in case the same configured source is used in two different connections. Also I have some concerns around the locking and updating this variable to make sure that the refresh-token-chain never get two concurrent updates which could potentially invalidate the refresh token chain.

Is anyone able to help with my query? Thanks!

Hi Everyone, I m just started with the journey of building my first custom connector 🙂 I am having struggle to connect to http://localhost:8000. -> this site can't be reached Any ideas ?

I'm creating a custom Rest API connector using UI builder tool. I have question regrading incremental sync support. Is it possible to generate a more sophisticated JSON payload using incremental sync form? Sample filter criteria that needs to be passed to the API - and last sync date needs to be injected here as well:

{

"filterCriteria": [

{

"field": "lastModifiedDate",

"operator": "AFTER",

"value": "2023-04-02"

}

],

"sortingCriteria": {

"field": "lastModifiedDate",

"order": "ASC"

}

}

bigquery denormalized is not creating the

additionalProperties

and instead creating 2 columns

event_data.type

event_data.additionalProperties

is this the expected behavior? can someone help me on this. when i tried same on snowflake, it produces

event_data

as json and one column. i am looking for same behavior on bigquery denormalized connector.

"event_data": {
      "type": ["null", "object"],
      "additionalProperties": true
    },

PS: i'm using some legacy airbyte instance and can not upgrade it atm.

Hi, I have created a custom transformation, however, I am receiving the error 'No module named dbt.adapters.mysql' Any ideas? Thank you!

I posted this a few weeks ago on discuss.airbyte.io but didn't get any responses: https://discuss.airbyte.io/t/how-to-manage-internal-id-for-dynamic-streams/4153 I have a connector where I'd like to refer to a stream by name ("foo"), but the underlying API is all based on IDs ("1234"). During discovery I can iterate over all objects and display the stream name. At that point I have the ID in hand ("1234") which would be used for all future API calls. But it looks like there is no way to store the ID, so all I have when someone selects the stream is the name ("foo"). This means that every time I have to iterate over all objects just to find the ID to then make the API calls I want. It's not the end of the world, but am I missing something? Is there any way take information found during discovery and store it alongside the name for later use? There is a "namespace" section but the documentation states that all streams returned from a source should share a namespace so I'm not sure that would be for general use.

Hey I got the low code yaml file working nicely, except for one problem, I want to add a transformation and I am not sure where I am going wrong. I added like this with the itention of each record also getting the streamslice.job_id it was collected from, but using the UI to test it, no records are getting this field added. What am I missing?

record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path:
            - data
      transformations:
        - type: AddFields
          fields:
            - path: job_id
              value: '{{ stream_slice.job_id }}'
        - type: RemoveFields
          field_pointers:
              - ["attributes", "first-name"]
              - ["attributes", "last-name" ]
      paginator:
        type: DefaultPaginator
        page_token_option:
          type: RequestPath
        pagination_strategy:
          type: CursorPagination

cursor_value: '{{ response[''links''][''next''] }}'

I have a custom data warehouse with me and a jdbc driver to run it. I want users of my source connector to be able to run sql operations using the source connector. How can I make this exactly using the Airbyte CDK?

I'd appreciate any sort of help i can get I am fully stuck in this

Getting this error using the Salesforce connector

2.0.9

Error: Cannot receive data for stream 'EmailMessage', error message: 'exceeded 100000 distinct ids'

Any suggestions on how to fix this, or backfill in chunks?

Does anyone know how to test the Jira connector? I'd like to fix this issue: https://github.com/airbytehq/airbyte/issues/24981 The source code change is a one-liner but I need help with tests. The unit tests are extremely basic. Just asserting that streams exist with no content checks at all, so there's nothing to change there (though seems like maybe they could/should be expanded). The bulk of the testing seems to be through integration tests that I think are connecting to some shared Jira instance. The README states:

**If you are a community contributor**, follow the instructions in the [documentation](<https://docs.airbyte.io/integrations/sources/jira>)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_jira/spec.json` file.
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.

However, that documentation contains nothing about generating credentials.

Could I get some guidance on building a source connector for my use case. I have a custom data warehouse with me and a JDBC driver to run it. I want users using my source connector to run SQL Operations against my data warehouse. How can I get started with building this using the Airbyte CDK?

I am fully confused about this step and I do not know where I can start from and would love to have some guidance

Hello team, I have created custom connector and on instance i have uploaded the docker image to get the new connector. Conenctor is working fine in local, and on interface as well. I can see emitted records as well. but when i look into the schema, the main table is blank. What could be the reason ? is this a version issue or am i missing something