Airbyte #ask-community-for-troubleshooting

Madison Mae

05/22/2023, 11:00 PM

Hi all, I am experience a

rate limit exceeded error

with my Hubspot connectors. It ended up fixing itself but oddly has only happened at the same time (841 844pm) the last two nights. Does anyone know why this keeps happening? And why only at this time? Is there a config I can change so this doesn't happen again?

yuan sun

05/23/2023, 1:18 AM

Hey all,I want to implement an API source where I can input the URL, Authorization, etc., and collect data from the API. When creating a connection, the

sources/discover_schema

endpoint needs to be called. Is it possible to dynamically generate schemas for different APIs? If so, how can I achieve that?

Johannes Müller

05/23/2023, 7:57 AM

I get duplicate entries with MySQL source which is configured as Incremental + Deduped history. I reset my data and still get duplicates:

Simon Thelin

05/23/2023, 8:05 AM

Is there anyone working on a trino destination? Or just a pure delta writer with for example

delta-rs

Axel Waserman

05/23/2023, 8:52 AM

Does the postgres source connector detects whenever new tables are added to the postgres source schema or does it only detect new columns within an already synced table ?

Axel Waserman

05/23/2023, 9:00 AM

Does Airbyte send a notification whenever a new postgres source table becomes available ?

Guy McCombe

05/23/2023, 10:20 AM

Hey - is there a way to have a no normalization (raw JSON) connection without having the

_airbyte_raw

prefix on each stream?

Simon Veerman

05/23/2023, 12:02 PM

Is it possible to run the initial sync for Mailchimp -> BigQuery on Open Source and then have Airbyte Cloud handle the daily updates in the same dataset?

Ton Badal

05/23/2023, 1:43 PM

Hello Airbyte community! I’m having problems with the Linkedin Pages connector. I was able to create and configure the source and create a connection, but when i try to access it or edit it from the UI i get this error, seems that

total_follower_count

and

share_statistics

are duplicated which triggers an error:

Copy code

Internal Server Error: Duplicate key io.airbyte.protocol.models.StreamDescriptor@46eb12d9[name=share_statistics,namespace=<null>,additionalProperties={}] (attempted merging values io.airbyte.protocol.models.AirbyteStream@2d4d38db[name=share_statistics,jsonSchema={"$schema":"<http://json-schema.org/draft-04/schema#>","additionalProperties":true,"type":"object","properties":{"totalShareStatistics":{"type":["null","object"],"properties":{"shareCount":{"type":["null","integer"]},"uniqueImpressionsCount":{"type":["null","integer"]},"clickCount":{"type":["null","integer"]},"engagement":{"type":["null","number"]},"shareMentionsCount":{"type":["null","integer"]},"likeCount":{"type":["null","integer"]},"impressionCount":{"type":["null","integer"]},"commentMentionsCount":{"type":["null","integer"]},"commentCount":{"type":["null","integer"]}}},"organizationalEntity":{"type":["null","string"]}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}] and io.airbyte.protocol.models.AirbyteStream@6b62fc6d[name=share_statistics,jsonSchema={"$schema":"<http://json-schema.org/draft-04/schema#>","additionalProperties":true,"type":"object","properties":{"totalShareStatistics":{"type":["null","object"],"properties":{"shareCount":{"type":["null","integer"]},"uniqueImpressionsCount":{"type":["null","integer"]},"clickCount":{"type":["null","integer"]},"engagement":{"type":["null","number"]},"shareMentionsCount":{"type":["null","integer"]},"likeCount":{"type":["null","integer"]},"impressionCount":{"type":["null","integer"]},"commentMentionsCount":{"type":["null","integer"]},"commentCount":{"type":["null","integer"]}}},"organizationalEntity":{"type":["null","string"]}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}])

anyone has successfully configured linkedin pages connector recently? Running on linux machine, airbyte version

0.44.3

, linkedin connector

1.0.1

Octavia Squidington III

05/23/2023, 1:45 PM

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-bootloader At 16:00 CEST / 10am EDT click here to join us on Zoom!

Valentin B.

05/23/2023, 2:59 PM

Hi everyone 👋 I tried the Databricks destination with postgres as a source and i'm a bit confused, it looks like airbyte jsonify all postgres table columns into

_airbyte_data

i only have these columns

Copy code

_airbyte_ab_id | string
_airbyte_data | string
_airbyte_emitted_at | timestamp

did i do something wrong? (maybe on the destination config but i follow connector doc and airbyte recommandations)

✅ 1

Lior Lev-Tov

05/23/2023, 3:49 PM

I created a connection between the salesforce source and Snowflake as a destination. after the sync, I figured out that the created tables in Snowflake do include the columns and data but don't include the relations between the tables. is there a way to include that as well?

Antoine Balliet

05/23/2023, 6:53 PM

Hi all ! I am using incremental sync using HubSpot connector to replicate data in BigQuery : companies stream for example. What is the best way to perform automated full-refresh (reset the cursor and pull all over again the data) once a week ?

Nohelia Merino

05/23/2023, 9:36 PM

Hello, I'm running into an issue while creating a new connector

Copy code

2023-05-23 19:31:29 [1;31mERROR[m i.a.s.a.ApiHelper(execute):28 - Unexpected Exception
java.lang.IllegalStateException: Get Spec job failed.
	at com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[guava-31.1-jre.jar:?]
	at io.airbyte.server.converters.SpecFetcher.getSpecFromJob(SpecFetcher.java:14) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.handlers.SourceDefinitionsHandler.getSpecForImage(SourceDefinitionsHandler.java:323) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.handlers.SourceDefinitionsHandler.sourceDefinitionFromCreate(SourceDefinitionsHandler.java:221) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.handlers.SourceDefinitionsHandler.createCustomSourceDefinition(SourceDefinitionsHandler.java:208) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.lambda$createCustomSourceDefinition$0(SourceDefinitionApiController.java:31) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:18) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at io.airbyte.server.apis.SourceDefinitionApiController.createCustomSourceDefinition(SourceDefinitionApiController.java:31) ~[io.airbyte-airbyte-server-0.40.23.jar:?]
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
2023-05-23 19:31:29 [1;31mERROR[m i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception

Graham DeNure

05/23/2023, 11:26 PM

Has anyone successfully set up S3 as a source? I get a 502 error on the

/api/v1/scheduler/sources/check_connection

route when I try and test the source.

Nipuna Prashan

05/24/2023, 12:15 AM

I have a data in the database column like 0x47326255F46785C89D1F7DA. However, when I sync it to the stage, it gets change to a different character set like G2bU�g�ȝ}��L�)�X�W{ what is the reason for that?

Mathias Klingler

05/24/2023, 9:37 AM

Hi all I have a 'failed to fetch' error if i want to create a connection (see picture). If I do a port-forward for the 'airbyte-v43-airbyte-webapp-svc' and create the same connection via localhost, then the connection can be established successully. Do you have any ideas? We user nginx as ingress.

Mathias Klingler

05/24/2023, 9:37 AM

image.png

Johannes Müller

05/24/2023, 10:17 AM

I would like to help testing a change: https://github.com/airbytehq/airbyte/pull/25859 I upgraded airbyte to the latest version 0.44.5
but it still seems to be using the old connector

airbyte/source-mysql:1.0.21

instead of

2.0.22

. Am I missing something?

Yusuf Mirkar

05/24/2023, 3:59 PM

why in destination postgres, while syncing, _airbyte_somename schema was created, what is that ? I had provided schema name as somename only

Yusuf Mirkar

05/24/2023, 4:34 PM

suppose sync starts at 12:00 and ends at 12:30 the records which were created in source on 12.20 will they be synced in same sync or next sync ?

Antoine Balliet

05/24/2023, 5:45 PM

Did anyone found an automated way to schedule full refresh on incremental sync ? -> This would be useful to reduce data inconsistencies over time when a lot of records are synced I thought about resetting cursor using SQL. Is there a smarter way ?

Yusuf Mirkar

05/24/2023, 6:18 PM

what is this - Failure Origin: replication, Message: Something went wrong during replication ?

Yusuf Mirkar

05/24/2023, 6:36 PM

what is airbyte-temporal ?

Luis Gomez

05/24/2023, 6:38 PM

Hello. I want to upgrade some connectors, but their associated connections have

LEGACY

states. Is there any standard way of migrating connection states from

LEGACY

STREAM

Octavia Squidington III

05/24/2023, 7:45 PM

🔥 Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-db + Airbyte Database Internals At 1PM PDT click here to join us on Zoom!

Yusuf Mirkar

05/25/2023, 6:24 AM

job_main_container_memory_request

env variables denotes the memory for source and destination worker containers and not the airbyte-worker container , right ?

Yusuf Mirkar

05/25/2023, 6:58 AM

how much memory to give temporal worker and airbyte-worker, airbyte-server ? not talking about source and destination worker container by job_main_container_* envs

Yusuf Mirkar

05/25/2023, 7:02 AM

what is airbyte-db used for during sync ?

Yusuf Mirkar

05/25/2023, 9:50 AM

what does this mean ? Note that all Source database connectors are Java connectors. This means that users currently need to over-specify memory resource for Java connectors.