Hey Y’all! I’m learning Airbyte and trying to set up microsoft Ads and it keeps failing. has anyone set it up successfully in the last 2 months?

I am trying to connect my custom source connector in UI. After selecting my connector, the below API fired and throws the following error. I have mentioned documentation_url in spec.yaml file. Can anyone help me how to resolve this error? http://localhost:8000/api/v1/source_definition_specifications/get

{
  "message": "Internal Server Error: Cannot invoke \"java.net.URI.toString()\" because the return value of \"io.airbyte.protocol.models.ConnectorSpecification.getDocumentationUrl()\" is null",
  "exceptionClassName": "java.lang.NullPointerException",
  "exceptionStack": [
    "java.lang.NullPointerException: Cannot invoke \"java.net.URI.toString()\" because the return value of \"io.airbyte.protocol.models.ConnectorSpecification.getDocumentationUrl()\" is null",
    "\tat io.airbyte.server.handlers.SchedulerHandler.getSourceDefinitionSpecification(SchedulerHandler.java:283)",
    "\tat io.airbyte.server.apis.ConfigurationApi.lambda$getSourceDefinitionSpecification$20(ConfigurationApi.java:405)",
    "\tat io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:886)",
    "\tat io.airbyte.server.apis.ConfigurationApi.getSourceDefinitionSpecification(ConfigurationApi.java:405)",
    "\tat java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)",
    "\tat java.base/java.lang.reflect.Method.invoke(Method.java:578)",
    "\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)",
    "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)",
    "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)",
    "\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)",
    "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)",
    "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)",
    "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)",
    "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)",
    "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)",
    "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
    "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
    "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
    "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
    "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
    "\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
    "\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)",
    "\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)",
    "\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
    "\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
    "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)",
    "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)",
    "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
    "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)",
    "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
    "\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
    "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
    "\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
    "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
    "\tat org.eclipse.jetty.server.Server.handle(Server.java:501)",
    "\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)",
    "\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)",
    "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)",
    "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)",
    "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
    "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
    "\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)",
    "\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)",
    "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)",
    "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)",
    "\tat java.base/java.lang.Thread.run(Thread.java:1589)"
  ]
}

Hello) Does Airbyte VAULT support IAM_AWS authentication method currently?

Hi, I’m new to Airbyte and am going through the documentation to find how to provision a custom code pipeline (e.g. something org specific web scraping) in a container that is executed by Airbyte. I’m looking at the section of building custom connectors but I have doubts that I’m reading the right sections. Any suggestions on Airbyte best practices how to do that?

hey guys. when i run a repliacton from oralce to elasticsearch ,i got an error.

Hi, I need help about a weird error message I get from Airbyte (probably since I've updated the MySQL plugin), any idea how I can fix it ? • I've also upgraded Airbyte to the latest version (0.40.6) • MySQL Source plugin version (0.6.12) It happens when I check my connection or when I try to synchronize, I don't have any issues with my other sources. Also, it'as about 2 different MySQL databases

Verification error(s) occurred for CHECK. Errors: [$.replication_method: should be valid to one and only one of the schemas , $.replication_method: string found, object expected]

Good day! Im trying to make a connector which uses Basic Authentication or Cookie-based Auth (I’m not sure which of the two it is). It’s the first time that I am making an Airbyte connector and I am having trouble coding this connector. The docs don’t provide a lot of information how basic auth works within the Airbyte CDK. I currently keep getting a

401 Client Error: No authorization data present for url

Eventhough the link is correct and the given username and password are correct. Anybody an idea how I should code a Airbyte Basic Auth connector or what could be the problem?

Hey folks, Replicating Freshdesk data with Airbyte Cloud. Appears to be going well however the “tickets” dataset has not replicated it’s full complete history. I’m not sure why. Rate limits are set so that the connector should control how often it hits the Freshdesk API. I didn’t set an “updated-at” property when setting up the connector to limit when the replication should start. Does anyone who has experience with this connector able to shed some light on how it works?

Hey all, Their appears to be a bug in the latest version of airbyte where the only tables getting picked up in postgres cdc are what is in the publication. This also means views can't be synced. How do I downgrade airbyte?

Looks like it's not a bug, this is an intended feature? How do we sync views and CDC together? https://github.com/airbytehq/airbyte/pull/14447

Hi all, quick question, is there a way to optimize airbyte fecth data from source? I'm currently extracting data from mysql with cdc option enabled, the table has 285GB, and the process is already running about 36 hours. My connection is mysql source to bigquery destination (using gcs to avro)

for me looks that chunks is did in small parts, I saw that we can increase workers, but don't know is it the case.

Hey folks, trying out airbyte for the first time. Following this document https://airbyte.com/connections/HubSpot-to-Redshift to setup a hubspot to redshift connection. Was able to setup the source and destination right. However, each time I go to sync it says successful without any bytes or records moved

Hi, All of my connection in Airbyte failed due to following reasons Source: SQL sever/Googlesheet Sink: BigQuery Does anyone know how to solve it. I am using airbyte cloud

Hey team, completely new to airbyte and been trying to set it up with K8s & Helm following official guide. Got random errors here and there, tried to play with component versions with no luck. Looks like 0.40.5 has a lot of fixes and will unblock us. Any ETA when chart will be published?

Hi guys, I'm trying to setup a destination using an on-prem MySQL DB, I keep getting this error message

```The connection tests failed.

Could not connect with provided configuration. HikariPool-1 - Connection is not available, request timed out after 60003ms.```

Can anyone guide on my setting up the destination? PS: I was able to successfully replicate data from the affected DB to a PostgreSQL DB (on-prem as well)

Hello Guys, I am totally new to Airbyte, Can someone help me in building connectors to API?

Hello Everyone, I have requested new connector, when can I expect this request to be validated, any idea?

Hello. I am currently trying to upgrade to the latest airbyte release. I have been using airbyte to sync from postgres to s3. Postgres source still works. But when I try to setup my bucket destination, it gives me 403 error, even though I know I am providing the correct creds. Airbyte logs mentions that the bucket does not exist, but it does. However, the bucket name has a digit in the beginning, in this case

7b-lakehouse-dev

. Is there a chance that in the latest s3 destination it does not like the

digit

as the first letter in the bucket name?….

Hey, why can't I select a cursor field here ? During sync it crashes telling me I have to select a cursor field but I can't 😅

Hey, • I am trying to build a custom connector. I have followed the steps and built a source http connector using pokemon api • When I tried to get the response from an api I got this error

{"type": "LOG", "log": {"level": "FATAL", "message": "1 validation error for ConfiguredAirbyteCatalog\nstreams\n  field required (type=value_error.missing)

• I did not change anything in the

configured_catalog.json

that I downloaded from the documentation any idea how to resolve this?

Hi, I cant delete sources or connections. When I delete sources I have a popup with a massive error message. When I delete a connection, on refresh it come back 😕

👋 Hello, team! I tried to create a connection between mongodb and postgres database and I got this error 2022-09-15 112814 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - com.mongodb.MongoCommandException: Command failed with error 40353 (Location40353): 'Invalid $project :: caused by :: FieldPath must not end with a '.'.' on server. The full response is {"ok": 0.0, "errmsg": "Invalid $project :: caused by :: FieldPath must not end with a '.'.", "code": 40353, "codeName": "Location40353"} what could be the issue here

Hi Team, Any idea, why I am seeing only hours not minutes in the below[Replication Frequency]? AirByte Version: 0.40.4

Hello, 1st congrats for the great tools that is Airbyte, I like it a lot ! At my client company, we are evaluating new ingestion tooling for our stack. I’ld love to use Airbyte. But one of the challenger in the decider team says that one very important feature is “handling schema automatically” (which another challenger tool is doing great - I won’t name it 😉 ). On my side, I want to push for Airbyte and I have checked a lot resources on that, especially that it’s on the roadmap for Q4: https://app.harvestr.io/roadmap/view/pQU6gdCyc/airbyte-roadmap • But the challenger says that the roadmap is not a given at all, especially that already in 2021 it was already state for Q4: https://airbytehq.slack.com/archives/C019WJFR1JT/p1640845210130100 Also the links about issues on the topic, looks not to move since several semesters: • https://discuss.airbyte.io/t/how-does-airbyte-handle-replication-when-a-data-source-changes-its-schema/90/2 • https://github.com/airbytehq/airbyte/issues/1967 Does anybody has more information and visibility on what happen on this feature ? Thanks much in advance Cheers all

Hi ! Does someone knows how to fix the error of synchronization (MySQL <-> BigQuery) error : " Pickling client objects is explicitly not supported. "

Hi all, looking for a bit of advice regarding data ingestion. (edit: not native english so hope it makes some sense) I correctly set up a shopify to bigquery connection. I usually prefer to use the bigquery dts destination to have a single table, however I cant find incremental + dedup option, so, to avoid to dedup data later I could do a full ovewrite at each refresh. OR I could use the bigquery destination, set up incremental + dedup, but I would find myself with dozens of shopify tables in my bigquery table. My question is: • should I am for a clean bigquery table and go for full overwrite? • should I aim for incremental + dedup, and nevermind if I have to scroll on dozens of tables in bigquery. Sync is once every 24 hours, shop is not massive (live 40MB of data from the sync data).

Hello, Did the docker installation and airbyte download like earlier but giving below error like while running docker-compose up

ERROR: Invalid interpolation format for "environment" option in service "worker": "CONFIG_DATABASE_PASSWORD=${CONFIG_DATABASE_PASSWORD:-}"

Hi guys, I am trying to transfer data from oracle (source) to postgres (destination) in incremental append mode but it does not show option for same

How to call Airbyte jobs from HTTP call? or n8n if anyone has done that