Hey everyone, We just release Clazar source integration, wanted to know if there is any observability metrics that we can have on the users who use our integration? if it's broken for any user using our integration? cc: @Aditya Khokhar

Hi! 👋 We're really looking forward to use the Sharepoint connector but I have a few questions, does it support Pages and Lists? Looking at the docs it seems to sync only the file system

Hi there, I’m trying sync google analytics 4 data by using custom report. But after inputing all dimensions and measures needed, I can not change the options in three button at the end of custom form and still can not click Done (because that button is not highlighted). What is the problem and how I fix it? Hope you guys will help me find out the solution. Many thanks.

This message was deleted.

Hi , I am getting an error when trying to build a custom connector that run a simple GET command from an internal REST API . this is the errors i am getting - I checked the API request on Postman and it works + we checked networking between the Aribyte Server & the API . Can someone give me a direction / help of what should I do next ? this is the error I am getting (does not help a lot)

Something went wrong in the connector. See the logs for more details. - Traceback (most recent call last):
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 135, in read
    yield from self._read_stream(
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 231, in _read_stream
    for record_data_or_message in record_iterator:
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 174, in read
    for record_data_or_message in records:
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 128, in read_records
    yield from self.retriever.read_records(self.get_json_schema(), stream_slice)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 375, in read_records
    for stream_data in self._read_pages(record_generator, self.state, _slice):
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 298, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 484, in _fetch_next_page
    return self.requester.send_request(
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 294, in send_request
    request, response = self._http_client.send_request(
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 331, in send_request
    response: requests.Response = self._send_with_retry(request=request, request_kwargs=request_kwargs, log_formatter=log_formatter)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 209, in _send_with_retry
    response = backoff_handler(user_backoff_handler)(request, request_kwargs, log_formatter=log_formatter)  # type: ignore # mypy can't infer that backoff_handler wraps _send
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 291, in _send
    raise UserDefinedBackoffException(
airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: The request failed due to an unknown error.
ERROR
None - Traceback (most recent call last):
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 302, in _read_stream
    yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, state)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 166, in read
    for message in self.source.read(self.logger, config, catalog, state):
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 167, in read
    yield from super().read(logger, config, catalog, state)
  File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 185, in read
    raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)
airbyte_cdk.utils.traced_exception.AirbyteTracedException: None

@Natik Gadzhi any insights you could provide here ? Thanks 🙂

Hi All. Can someone help me deleting a custom connector that is already published? Thanks in advance!

I'm using the nocode connection builder and i'm using the session token authentication method. Does anyone know how to reference the Session Token (pulled from the Session Token Path) as a variable or something? I'm trying to use the variable in the body of the other stream requests and the ui only has the option for Injecting the token into the Header or Query when using API Key for Data request authentication.

Hey all. Thank you for your amazing product. I'm using the nocode connection builder to call my api and everything works exept for the stream state management: I have configured the Incremental Sync by selecting the Start Filtering capability and injected the state datetime in the json payload. The api returns what i expect:

[
  {
    "DeclaredQuantity": 0,
    "EventTime": "2024-07-30T17:38:47.52"
  },
  {
    "DeclaredQuantity": 0,
    "EventTime": "2024-07-31T10:31:54.843"
  },
  {
    "DeclaredQuantity": 0,
    "EventTime": "2024-07-31T10:58:12.163"
  }
]

What i expect is that the Stream state is set to the last EventTime (

2024-07-31T10:58:12.163

in this case) but it seems that the state can be set only to datetimes starting from yesterday

[
  {
    "type": "STREAM",
    "stream": {
      "stream_descriptor": {
        "name": "Incremental"
      },
      "stream_state": {
        "EventTime": "2024-07-30T17:38:47.52"
      }
    },
    "sourceStats": {
      "recordCount": 3
    }
  }
]

Another intresting thing is that if i set the Starting Datetime (configured to be a user input) to today, the API returns data correctly but the stream state is not set at all:

[
  {
    "type": "STREAM",
    "stream": {
      "stream_descriptor": {
        "name": "Incremental"
      },
      "stream_state": {}
    },
    "sourceStats": {
      "recordCount": 2
    }
  }
]

Does anyone have suggestions? Thanks in advance for your help!

My PR has been in review for many days, can someone take a look at it? https://github.com/airbytehq/airbyte/pull/42851

So while creating a stream for a Custom connector of Facebook Pages I´m facing an issue and it´s the following on the image below I just need to get everything inside the "Data" to be a unique column for each field, how can i do that Thanks!

Hey, I'm trying to use the Low-code cdk to develop a connetor. I'm in the initial steps. I finished cloning the repo, and whenever i try to run the ./generate.sh command, the interactive helper does not pop up for me. instead, the command makes a docker build called "generator" and that's about it. how can i fix this so that the interactive helper shows up, and I can move on with my build?

Would this Bigquery destiny problem that was fixed on 2.8.4 hand all other connections?

Hi everyone. I am getting this error in a EC2 local deployment when testing a source, I even tried to set the latest version of airbyte-connector-builder. Could someone help me?

The manifest version 2.3.0 is greater than the airbyte-cdk package version (2.0.0). Your manifest may contain features that are not in the current CDK version.

I'm using the airbyte low-code cdk. My API requires weird authentication. 1. I first have to request an authentication endpoint with 4 parameters in the payload of my api request (username, password, developer key and organization id). I then get a response that has a sessionId. 2. I need to then use the sessionId along with the developer key within the payload of the api request to any endpoint I want. is this possible using the low-code cdk? if it is, i'd really appreciate some help on how I could set this up. Thank you

Hi, is it possible to fetch substream data in parallel to speed it up? I have an incremental parent stream which has a primary key of “fileid”, I have a substream which can fetch the parent details path using that same “fileid” as the parent. The issue is that the details page is one at a time so I’m trying to run e.g. 10 parallel requests to get details. I am trying to use the low code solution for this. Thanks!

Hey all, how I can set a pinecone namespace on the pinecone destination? https://docs.pinecone.io/guides/indexes/use-namespaces#creating-a-namespace

I have a more complex ETL jobs that also presumes a minimal process of the data, preferably in Python. I need to get data from multiple SQL schemas and tables. Out of them multiple output tables will be resulted. Is not only a transform, I need to do also fan ins and aggregates. Few examples * from source: 3 related tables, I need to extract some data stored in their JSONs properties. The output will be insertion into 5 tables after some python processing. I presume it could be done as a source connector, that manipulates the data, and the output schema is the same as my destination schema I already have. I have a hard time to make a plan how to do it in Airbyte, I have managed to create simple replication rules already, but I have not saw any complex source connector to understand how would I do that, any feedback would be helpful.

Hello! Does anyone know of a source connector that supports incremental sync but does not use a date- or datetime-based cursor? I am working on changes to the Plaid connector to support Plaid's transaction sync, and Plaid uses random strings as cursors. There is no datetime field that can be used as a data cutoff. Instead, Plaid expects you to store the last cursor value it returned and use that value the next time you sync. Is it possible to implement incremental sync via Airbyte by persisting a random string as a cursor between syncs? If it's impossible, it would be nice to know before sinking a ton of time into trying to make it work. :)

Hi guys, Im a student and I wanna know if there is a way to sync my postgres (from Supabase) database with Meilisearch ? There is no destination for meilisearch 😅

I'm trying to connect to Azure Synapse db using the mssql connector but I hit the schema discovery issue: https://github.com/airbytehq/airbyte/issues/42491. Would it be possible to provide airbyte a custom, manually created source catalog for the existing connection so that I can enable/disable streams and read data using airbyte?

Hello community! Is

DeclarativeCursor

, e.g. to use datetime stream slicers for cursor fields still supported for low-code connectors? Thanks 🙏

Hello, I’m using Airbyte Cloud to create custom connectors through their UI Builder. I’m working on a simple connector that makes an API call and receives a JSON response. I want to implement incremental sync using a timestamp as the cursor. The response data includes a

timestamp

column in the format

%Y-%m-%d %H:%M:%S

, and the API accepts

from

and

to

parameters in the format

%Y-%m-%d

. For example: API:

<https://api.test.com/xyz?from=2024-08-10&to=2024-08-10>

Response Data:

[
    {
        "timestamp": "2024-08-06 00:11:55",
        "timezone": "UTC"
    }
]

How can I configure the connector to support incremental sync using the

timestamp

as the cursor and ensure that the

from

and

to

parameters in the API URL are set based on the stream state (using the last execution date) up to yesterday’s date?

Hi there, I am evaluating Airbyte for use in our tech stack and have tested out creating my own connector for an API source. I want to know though, if the Connector Builder is also used for creating a connector for a REST API as the destination? So, not pulling data from the API source, but pushing data from a source to the API as the destination. I can't find anything about it the docs, so not sure if this is possible? The only reference is to API sources.

Hi, I learned that Airbyte expects already structured data for its connectors. What would be a platform be for scraping/crawling multitude of sites. I am working on a research project where we want to integrate about 10k of different source sites that publish pdf/word/html into a common search backend for the german cities and counties. I know I can use various scrapping tools, but given the multitude, I feel I need a platform in its own right to configure, maintain and monitor these scrapers, also allow different people to contribute and extend/fix them? Any ideas or pointers are welcome. Current best idea: Browserless -> Airbyte -> CKAN

Hi im using the low code cdk. I want to implement incremental sync in such a way that only new data from my api is synced into my postgres database. how exactly am I supposed to go about doing this? My api supports filtering based on createdTime and updatedTime field.

Hi Team, I am trying to buid the no-code custom connector in airbyte I have successfully created the parent stream and applied pagination as well below is the screenshot for parent stream. Parent stream: After this I needed to create the substream and pass the id which is the result from the parent stream to the substream. but there I am getting the below error: ERROR

Request to <https://pi.pardot.com/api/v5/objects/emails?fields=id,clientType,prospectId,listEmailId&nextPageToken=eyJvcmRlckJ5IjoiIiwiZmlsdGVycyI6W10sImxpbWl0IjoyMDAsInJlc3VtZVZhbHVlIjp7ImlkIjo0ODc3MTk4N30sInBhZ2UiOjEsInJlY0NvdW50IjoyMDAsImV4cGlyZVRpbWUiOiIyMDI0LTA4LTEzVDEwOjU2OjEyLTA3OjAwIiwiZGVsZXRlZCI6bnVsbH0=> failed with status code 400 and error message Page token has expired. - Traceback (most recent call last):

Hi all, I am trying to connect my Google Drive as a Source in Airbyte but facing some issues and not able to resolve it. 1. I have tried it with Service Account Key Authorization as well, via giving the access to the mail of service account still not able to configure it properly 2. Tried it with Google OAuth still facing the same issue, how to resolve it. If anyone knows about it please help!

Hello everyone! I’m working on a custom connector and could use some help. I’m super new to this and eager to learn. I’m learning this on the fly so your guidance is much appreciated! I need to do two things in sequence: 1. Send a POST request to kick off an async export, which gives me an

export_id

. 2. Use that

export_id

in a GET request to download the export. I’m stuck on how to grab the

export_id

from the POST response and use it in the GET request. This is what it looks like on the records after the POST request. Any tips or examples on how to handle this? Let me know if you’d like any more info. Thanks a bunch!

Connector builder with OAUTH2 I am trying to build a connector for an API with OAUTH2, the auth endpoint of the API returns the token in a nested format, does anybody know how I can parse this? In the builder only a

Access Token Property Name

can be defined. Ideally I would like something like this (but the response object is not available):