Is there a way to customize just a little bit the authentication process in the visual connector builder? The API I'm trying to build the connector for uses OAuth2.0 but it need a Client ID, Client Secret and a Basic Token, all of them are provided, but i can only fill the fields for Client ID and Client Secret. After this authorization, I get an access_token, which expires and i have to request a new one in the above flow. I tried to figure something to do with the yaml file but didn't manage to accomplish anything. There's some way of doing this way or do i need to do it in the python cdk?

This message was deleted.

how to upload new connector

Hello, I am using your connector builder solution to build a connection between pagar and snowflake (thanks a lot it is great), the issue is that one object of pagar can only be read with a timestamp in milliseconds and it seems like it is not possible to use a date format such as %s in your connector builder, am I right ?

Hello, I’m building a REST API-based custom connector and what to ask this question In the function

parse_response

, I need to return the data from API as the data while this connector read from source I want to ask how can I access the json_schema where the source schema is being defined in this function

Hello 👋 Here's a summary of the Connector Builder features we released over the last week (06/26 - 07/02) - Support incremental syncs with cursor field sorted in descending order - Allow reverting to a previously published connector version - Allow specifying plain text request bodies - Allow composite primary keys - 🐛 Support datetime formats that are not lexicographically sortable Be sure to upgrade to the latest version to benefit from these improvements! Airbyte platform: v0.50.6 CDK: 0.44.2

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

I have a connector for Personio that I have built as a Python CDK connector - although only basic support for the streams so far - however this connector would work with the normal custom connector process, except that it has a different authentication flow. Is there a way to create a custom auth only, or would you accept a general feature to Airbyte that supports this flow? it is an auth endpoint on the API that generates a single use bearer token. Each response when paginating includes the next single use bearer token.

Hello team!. I'm building my destination-mysql connector custom. It's working for me, but I have trouble when It's not support normalization. So how to add normalization to my connector?

In my python source connector, the log message for my state emissions is large because my state is large (contains a large array). What is the best way to avoid large log lines here? Can I customize the logger message formatter?

Explain to me about state.json file for the python cdk

Why is develop locally removed from tutorials? https://github.com/airbytehq/airbyte/blob/master/docs/contributing-to-airbyte/developing-locally.md I am trying to write simple jdbc java source connector for databricks. I was following above document. Are there any updates in it which we should be aware of? I am facing difficulties in running gradle build. Some acceptance tests are failing which i have not even touched. Log of gradle build:

E               requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
         
         .venv/lib/python3.9/site-packages/requests/adapters.py:578: ReadTimeout
         
         
          test_utils.py::test_not_found_container ???                        98% ??????????????????????????????
          test_utils.py::TestLoadYamlOrJsonPath.test_load_json ???           98% ??????????????????????????????
          test_utils.py::TestLoadYamlOrJsonPath.test_load_yaml ???           99% ??????????????????????????????
          test_utils.py::TestLoadYamlOrJsonPath.test_load_other ???          99% ??????????????????????????????
          test_utils.py::test_find_all_values_for_key_in_schema[schema0-looking_for_this_key-expected_values0] ???99% ??????????????????????????????
          test_utils.py::test_find_all_values_for_key_in_schema[schema1-looking_for_this_key-expected_values1] ???99% ??????????????????????????????
          test_utils.py::test_find_all_values_for_key_in_schema[schema2-looking_for_this_key-expected_values2] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_discovered_catalog_and_empty_streams[discovered_catalog0-empty_streams0] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_discovered_catalog_and_empty_streams[discovered_catalog1-empty_streams1] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_custom_catalog[custom_configured_catalog0-False] ???99% ??????????????????????????????
          test_utils.py::test_build_configured_catalog_from_custom_catalog[custom_configured_catalog1-True] ???100% ??????????????????????????????=============================== warnings summary ===============================
         test_config.py::TestConfig::test_config_parsing[Legacy config should be parsed without error.]
         test_config.py::TestConfig::test_legacy_config_migration[A legacy raw config is parsed into a new config structure without error.]
         test_config.py::TestConfig::test_legacy_config_migration[A legacy raw config, with a test_strictness_level defined, is parsed into a new config structure without error.]
           /Users/ameya/Workspace/airbyte/airbyte-integrations/bases/connector-acceptance-test/connector_acceptance_test/config.py:285: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
             logging.warn("The acceptance-test-config.yml file is in a legacy format. Please migrate to the latest format.")
         
         -- Docs: <https://docs.pytest.org/en/stable/warnings.html>
         =========================== short test summary info ============================
         FAILED unit_tests/test_utils.py::test_docker_runner[standard] - requests.exce...
         FAILED unit_tests/test_utils.py::test_docker_runner[waiting] - requests.excep...
         FAILED unit_tests/test_utils.py::test_not_found_container - requests.exceptio...
         
         Results (624.33s (0:10:24)):
              543 passed
                3 failed
                  - test_utils.py:268 test_docker_runner[standard]
                  - test_utils.py:268 test_docker_runner[waiting]
                  - test_utils.py:308 test_not_found_container
         /Users/ameya/Workspace/airbyte/airbyte-integrations/bases/connector-acceptance-test/.venv/lib/python3.9/site-packages/coverage/control.py:788: CoverageWarning: No data was collected. (no-data-collected)
           self._warn("No data was collected.", slug="no-data-collected")

> Task :airbyte-integrations:bases:connector-acceptance-test:_unitTestCoverage FAILED

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

hello there, i am trying to create a new source via a docker image, but got an error: Fetching connector failed. Try again i have tried the suggestions here https://airbytehq.slack.com/archives/C027KKE4BCZ/p1687419962093719but still no luck...

Hello, I'm using destination mysql, but I have trouble "Waiting for schema metadata lock" when exec "CREATE SCHEMA IF NOT EXIST". How to fix it?

Hi everybody. I am currently using AirByte Open source. I want to create a login account for users with access rights, how do I create it?

how to change default account and password of AirByte Opensource?

This message was deleted.

Hello Everyone, I have build a connector using the builder UI to read and extract data from an HTTP API. Where i have different API’s to be done. When i the give the api explicitly like for example

<https://registre-national-entreprises.fr/api/companies/diff>

i got the records as a json

[
  {
    "company": {
      "updatedAt": "2023-07-05T01:28:21+02:00",
      "id": "64a4ab15779d4bbf370ceb23",
      "formality": {
         }
      }
  }
]

Some data like this and its working well. ** I have some other API where i should give the value implicitly and read the id (Because i have lot of id’s) and extract the data.

<https://registre-national-entreprises.fr/api/companies/{{> config['id'] }}/attachments

this says the request failed. I hope someone might tried this earlier and please give suggesstions and solution. That would be really helpful. Thanks in advance !!

Hello Team, is anyone aware how we can get partial success status for particular sync? We tried to sync 200 MB of data from source to destination and then shutdown the DB at source, I was expecting to get status as Partial success, but instead I got failure. Anyone aware of any use case to generate partial success? I am using airbyte version 0.50.5

If my API returns a URL should i declare the type as "string" or something else in my schema?

Want to find that documentation where it explains about how incremental sync should be implemented and it should give one duplicate row so that we know that incremental sync is working

Good afternoon. I'm using the builder pointing to an api that extracts data from mongodb. In airbyte OSS, this ingestion works successfully by sending the data to google bigquery, but when I use airbyte cloud, it returns a kubernetes pod memory error in the ingestion log: Exit code 137. Could someone help me?

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1PM PDT click here to join us on Zoom!

Hi there, I have the Airbyte Cloud and created an custom connector that has been failing to reset. The message says something went wrong but it is not specific about the issue. I opened a ticket and was told to come here. Job id 2987000

Hi all, Can some one please let me understand what this error means? Thank you very much for your help in advance

2023-07-06 04:25:09 ERROR i.a.p.j.JobNotifier(notifyJob):138 - Unable to read configuration for notification. Non-blocking. Error:
java.lang.NullPointerException: Cannot invoke "io.airbyte.config.NotificationSettings.getSendOnFailure()" because "notificationSettings" is null
	at io.airbyte.persistence.job.JobNotifier.notifyJob(JobNotifier.java:114) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.4.jar:?]
	at io.airbyte.persistence.job.JobNotifier.notifyJob(JobNotifier.java:76) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.4.jar:?]
	at io.airbyte.persistence.job.JobNotifier.successJob(JobNotifier.java:212) ~[io.airbyte.airbyte-persistence-job-persistence-0.50.4.jar:?]
	at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.jobSuccessWithAttemptNumber(JobCreationAndStatusUpdateActivityImpl.java:285) ~[io.airbyte-airbyte-workers-0.50.4.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-07-06 04:25:09 INFO i.a.c.t.StreamResetRecordsHelper(deleteStreamResetRecordsForJob):50 - deleteStreamResetRecordsForJob was called for job 79 with config type sync. Returning, as config type is not resetConnection.

How can we be notified by the airbyte api that a sync is completed

Hello everyone! I'm exploring Airbyte for data integration, and I'm wondering if there is a Cassandra source connector available in Airbyte. I would like to pull data from my Cassandra database and integrate it with other systems. Could someone please let me know if there is an existing Cassandra source connector in Airbyte or if there are any plans to develop one? Thank you!"

I'm looking for an example using the python CDK for how to manipulate the data of a particular stream before it gets transformed in to a record. I want to transform the shape of the data, mapping the json to different result names, and simplifying the data?

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves