Good afternoon, while trying to upload a custom connector using a public docker image, on a local airbyte portal gui, I'm getting the following error logs, could someone help me out to understand those logs. Thank you

2023-05-23 19:31:29 [1;31mERROR[m i.a.s.a.ApiHelper(execute):28 - Unexpected Exception
java.lang.IllegalStateException: Get Spec job failed.
	at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.1-jre.jar:?]
	at io.airbyte.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:14) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.handlers.SourceDefinitionsHandler.getSpecForImage(SourceDefinitionsHandler.java:323) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.handlers.SourceDefinitionsHandler.sourceDefinitionFromCreate(SourceDefinitionsHandler.java:221) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.handlers.SourceDefinitionsHandler.createCustomSourceDefinition(SourceDefinitionsHandler.java:208) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.lambda$createCustomSourceDefinition$0(SourceDefinitionApiController.java:31) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:18) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.createCustomSourceDefinition(SourceDefinitionApiController.java:31) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-05-23 19:31:29 [1;31mERROR[m i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception

Hi, everyone. Can I send “parent_key” from the parent stream in manifest.yaml to the child stream in schemas/file_name.json? I miss the id of product_attributes_stream for the product_attribute_terms_stream in source-woocommerce. In the final table, after normalization, I can’t understand which line belongs to which attribute. The schema has no parent id because they don’t have this parameter in API: github-airbyte woocommerce-documentation However, the manifest.yaml has a link to its path: “/products/attributes/{{ stream_slice.id }}/terms” (github-airbyte) Can this parameter be added to the schema? Or do I need to use another way to add this parameter to the final table in the database?

Is it possible to override the schema of a source? For instance, can I manually define the schema for the tables of a postgresql database?

Hey Team, I was thinking of doing Hubspot integration using Airbyte. I was going through the docs and according to my understanding, using Airbyte I can only receive data from Hubspot but can't send data to it. 1. Am I correct? 2. If I am is there any plan to support the feature where Airbyte use will be able to send data to the CRM?

Hello everyone. I am trying to get started wit developing custom connectors but got stuck on pagination. I want to stream Missive's List Conversation Endpoint https://missiveapp.com/help/api-documentation/rest-endpoints#list-conversations but the pagination requires a timestamp. Any idea how i could workaround this? Thanks in advance!

Hello Airbyte friends! I am working on implementing an incremental sync for Shipbob and I am able to get the initial sync to work but the incremental syncs fail due to a mismatch in expected date formats. I get the following error on incremental runs

Additional Failure Information: time data '2023-05-22T20:36:11.6980713+00:00' does not match format '%Y-%m-%dT%H:%M:%S.%f+00:00'

Which I totally understand, the

created_date

from my API response is coming in with 7 digits for microseconds as opposed to the expected 6 digits but I am not sure how to handle this. Thank you in advance!

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals At 1PM PDT click here to join us on Zoom!

Hey Team, I have set up the Github connector by following the setup guide and it is working. But the data with stream name

releases

does not match the actual data. (We set the sync mode as

Incremental | Deduped + history

) Does anyone know why I am getting back the wrong data or what step should I take to solve this problem? Thank you.

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

This message was deleted.

What's the difference between the v0 and v1 classes of the protocol model?

Hey Team, I'm looking for an OAuth 2.0 compatible zoom source-connector. Airbyte supports jwt-compatible connector, which is deprecated by zoom. Can you provide me some pointers on how to get started? If there is any work-in-progress on this thread, I would love to be involved. Any pointers would be helpful.

Hello Team, I am trying to build AWS cost explorer connector using Airbyte CDK. Any idea on how to get started! Thanks

Need more info about this error. Getting this while make a connection between custom connecto kobotoolbox and postgres.

Caused by: io.temporal.failure.ApplicationFailure: message='java.lang.IllegalStateException: Duplicate key _PMU Monthly  Version  Sept (attempted merging values {"$schema":"<http://json-schema.org/draft-07/schema#>","additionalProperties":true,"type":"object","properties":{"endtime":{"type":["string","null"]},"_id":{"type":["number","null"]}}} and {"$schema":"<http://json-schema.org/draft-07/schema#>","additionalProperties":true,"type":"object","properties":{"endtime":{"type":["string","null"]},"_id":{"type":["number","null"]}}})', type='java.lang.RuntimeException', nonRetryable=false

Updating the docker image version for a connector keeps failing even though the image tag exists

how to change the docker registry for connectors from default airbyte

Loved the demo of the Custom Builder today. We upgraded our Airbyte instance to the latest release

0.44.4

but I am not seeing the “Builder” tab. What are we missing?

Hello, I am getting this error with the Builder Connector UI. Is there a way to resolve empty strings for a key? ERROR

Empty string keys not allowed without dpath.options.ALLOW_EMPTY_STRING_KEYS=True:

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

Hey Team! I'm trying to use the low-code config development process for a source connector. I right now need to cast types from my fetched data to the one I defined at schemas. https://docs.airbyte.com/connector-development/cdk-python/schemas#default-type-transformation I saw that this solves my problems! However, this guide is not so focused on low-code solutions (I guess) how should I use it? Building a custom

transformation

and setting inside the stream or building a custom

stream

? or may I somehow just use it on my

manifest.yaml

upfront without building custom code?

im building a custom connector in the CDK. to successfully use the incremental strategy, i need to pass a timestamp into the query params as a unix timestamp. however, the cursor field is a datetime string. i thought i could use jinja to manipulate the data type, but im getting an error in the connector.

time data '1621468800.0' does not match format '%Y-%m-%d %H:%M:%S.%f+00:00'

the CDK seems to break when the date types dont match between the cursor field and the query param.

I'm trying out the new Airbyte Connector builder UI. The API endpoint is https://chargify.com/transactions.json?per_page=200&direction=asc&order_by=id&since_id=XXX (GET), and a reduced version of the JSON payload from the API is shown below. I've configured the connection as Cursor Pagination with a custom next page cursor. I haven't figured out the magical combination of Record Selector and Cursor Value, and I've tried about 100. What I'm looking to do is grab the

id

value from the last fetched record, and use that in the request parameter

since_id=

. I think part of the problem is that the JSON is formatted kind of funny, such that the JSONPath to get to the id value is difficult. Anybody have suggestions?

Hi guys! Does anyone have ever developed a custom-connector for google-trends? Maybe using pytrends repo. We use google trends a lot in my company integrations and we are planning to start a connector. If anyone has any tip, let me know

Hi, guys! Is there any support plan for destination klaviyo?

I have some "feature wishes" for the connector Builder UI, and am not sure if this is the best place to register them. But here goes: 1. for pagination it should be possible to create a user input value for Page Size, rather than setting a static number in the UI. After all I might want 200 records per page but another person using the same connector might want 20. Plus a user input value could be used in a Stop Condition like this:

{{ last_records | count < {{ config['records_per_page'] }} }}

2. Incremental sync should allow for not just time-based cursoring but also integer/id-based cursoring. The API endpoint I've set up incremental sync for works a lot better if I can "scroll down" to the latest known ID rather than latest known timestamp. That's because this API only allows for starting from a specific date (format YYYY-MM-DD), not from a specific timestamp. So at best I can fetch everything since the last known date, which renders intra-day incremental sync repetitive/pointless and involves a lot of de-depuping. So it's best to simply wait until just after midnight and fetch the last 24 hrs worth of data.

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

Hi, Attended the airbyte connector builder session last week. I got the replay video but now I'm not able to open it. How to get access to it? Thanks

Might be the wrong place to log this! Newbie to the connector development. I've forked the latest version of the github repo -

v0.44.4

Trying to generate a HTTP connector - I went through the steps to from tutorial and getting this error message

While trying to generate a connector, an error occurred on line 38 of generate.sh and the process aborted early.  This is probably a bug.

Has there been an breaking change added?

Hello Airbyte friends, I am developing a custom connector. I'm using the latest version

v0.44.4

. I'm blocked on combining

Record Selector

and

Cursor Pagination

. I could not have all data from next pages, I got data from the first page only, if I set the

Record Selector

is

content

. In contrast, I will have all data thanks to

Cursor Pagination

but I could not use the result as parent stream for other sub streams. Do we have any way to combine them to get all data and useful for sub streams ? Could you help me please 🙏 You could refer my configuration file and the original data structure which ingested from API below.

Hi, I'm developing a destination connector for a to-be-published database. I can locally go through my connector along with a ./run-ab-platform.sh airbyte deployment, however I can not get normalization related option visible. I've tried as many ways as I count found, such as https://airbyte.com/tutorials/creating-duckdb-destination-with-python https://docs.airbyte.com/connector-development/testing-connectors/testing-a-local-catalog-in-development/, with no luck, and it seems some documents is out-of-date and some are rapidly changing. So is it possible to finish a locally end-to-end connector go through (w/ locally deployed airbyte, w/ normalization) before publish anything? Or I have to publish the connector first and then add normalization to it, like mentioned here https://docs.airbyte.com/connector-development/#4-publish-the-connector