Airbyte #ask-community-for-troubleshooting

Johannes Müller

03/21/2023, 7:01 AM

How can you filter out table columns in Airbyte? I know that it's possibe via a custom transformation. But the last comment here suggests that it is possible natively, yet I can't see that in the UI: https://github.com/airbytehq/airbyte/issues/9771 Maybe the issue was closed by accident?

Bismay

03/21/2023, 8:18 AM

Hi Team , I am able to create connection from local , mongodb to bigquery . but after creating ingress loadbalancer , source and destination is created successful but i am note able to create connection . Pasting the logs below - INFO 2023-03-21T074422.355828222Z [resource.labels.containerName: airbyte-server-container] 2023-03-21 074422 [33mWARN[m i.a.c.s.s.AirbyteGithubStore(getLatestDestinations):58 - Unable to retrieve latest Destination list from Github. Using the list bundled with Airbyte. This warning is expected if this Airbyte cluster does not have internet access. { "textPayload": "2023-03-21 074422 \u001b[33mWARN\u001b[m i.a.c.s.s.AirbyteGithubStore(getLatestDestinations):58 - Unable to retrieve latest Destination list from Github. Using the list bundled with Airbyte. This warning is expected if this Airbyte cluster does not have internet access.", java.io.IOException: Connection reset at jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:857) ~[java.net.http:?] at jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:123) ~[java.net.http:?] at io.airbyte.commons.server.services.AirbyteGithubStore.getFile(AirbyteGithubStore.java:83) ~[io.airbyte-airbyte-commons-server-0.40.32.jar:?] at io.airbyte.commons.server.services.AirbyteGithubStore.getLatestDestinations(AirbyteGithubStore.java:56) ~[io.airbyte-airbyte-commons-server-0.40.32.jar:?] at io.airbyte.commons.server.handlers.WebBackendCheckUpdatesHandler.getDestinationDiffCount(WebBackendCheckUpdatesHandler.java:60) ~[io.airbyte-airbyte-commons-server-0.40.32.jar:?] at io.airbyte.commons.server.handlers.WebBackendCheckUpdatesHandler.checkUpdates(WebBackendCheckUpdatesHandler.java:37) ~[io.airbyte-airbyte-commons-server-0.40.32.jar:?] at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:18) ~[io.airbyte-airbyte-server-0.40.32.jar:?] at io.airbyte.server.apis.WebBackendApiController.webBackendCheckUpdates(WebBackendApiController.java:62) ~[io.airbyte-airbyte-server-0.40.32.jar:?] at io.airbyte.server.apis.$WebBackendApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.40.32.jar:?] at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:378) ~[micronaut-inject-3.8.3.jar:3.8.3] at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.8.3.jar:3.8.3] at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:246) ~[micronaut-router-3.8.3.jar:3.8.3] at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.8.3.jar:3.8.3] at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.8.3.jar:3.8.3] at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.8.3.jar:3.8.3] at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.0.jar:3.5.0] at reactor.core.publisher.Flux.subscribe(Flux.java:8660) ~[reactor-core-3.5.0.jar:3.5.0] at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:426) ~[reactor-core-3.5.0.jar:3.5.0] at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.8.3.jar:3.8.3] at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2508) ~[reactor-core-3.5.0.jar:3.5.0] at reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:371) ~[reactor-core-3.5.0.jar:3.5.0] at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onSubscribe(ReactorSubscriber.java:50) ~[micronaut-runtime-3.8.3.jar:3.8.3] at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68) ~[reactor-core-3.5.0.jar:3.5.0] at reactor.core.publisher.Flux.subscribe(Flux.java:8660) ~[reactor-core-3.5.0.jar:3.5.0] at io.micronaut.core.async.publisher.Publishers.lambda$then$5(Publishers.java:324) ~[micronaut-core-reactive-3.8.3.jar:3.8.3] at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:67) ~[reactor-core-3.5.0.jar:3.5.0] at reactor.core.publisher.Flux.subscribe(Flux.java:8660) ~[reactor-core-3.5.0.jar:3.5.0] at io.micronaut.http.server.netty.RoutingInBoundHandler.handleRouteMatch(RoutingInBoundHandler.java:601) ~[micronaut-http-server-netty-3.8.3.jar:3.8.3] at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:457) ~[micronaut-http-server-netty-3.8.3.jar:3.8.3] at io.micronaut.http.server.netty.RoutingInBoundHandler.channelRead0(RoutingInBoundHandler.java:147) ~[micronaut-http-server-netty-3.8.3.jar:3.8.3] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:224) ~[micronaut-http-netty-3.8.3.jar:3.8.3] at io.micronaut.http.netty.stream.HttpStreamsServerHandler.channelRead(HttpStreamsServerHandler.java:134) ~[micronaut-http-netty-3.8.3.jar:3.8.3] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99) ~[netty-codec-http-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport- thanks in advance , Bismay Jena

Johannes Müller

03/21/2023, 9:04 AM

It is not possible to run

dbt deps

on the auto generated dbt project outside of airbyte since the path is hard coded to

/dbt

instead of (similar to the log files)

../dbt

since you usually don't have permissions to create files in the root directory. I added a fix for the mysql source. I can update it to follow best practices if you want to merg it but it's just a one line change, so not sure if it is worth the trouble :D https://github.com/airbytehq/airbyte/pull/24272

Fernand Ramat

03/21/2023, 9:14 AM

Hello, I am hosting Airbyte on EC2 instance (t2.large) / run it via docker-compose. I sometimes experience some issues when loading new source; this is kinda fine, but the problem is that the instance get completely blocked and not even accessible via ssh anymore. Is there is any possibility to restrict a bit the memory/cpu consumption ? Or should I directly check on the container/docker limitation directly? Thanks!

Johannes Müller

03/21/2023, 9:37 AM

Why do the auto generated models contain a

hashid

column with an md5 based on the row values? The inline documentation says

-- SQL model to build a hash column based on the values of this record

but I can see that much by reading the code :D

Will Truong

03/21/2023, 9:45 AM

Hi, I install Airbyte Open source in a remote server (Ubuntu). After run command on CLI, now how can I access to Airbyte UI in localhost? Because this server run in Ubuntu and it does not have browser. Airbyte works totally fine on my local machine, but when I try to implement in it the server and only use only command line, it starts to struggle complete.

Sebastian Brickel

03/21/2023, 9:53 AM

Hi team, I am working on a Microsoft SQL Server (MSSQL) <> BigQuery connection and get

Something went wrong in the connector. See the logs for more details

. I think I have a timeout problem, but I’m not sure, log file attached. Assuming I’m right, since I don’t have direct access to the MSSQL server, I’m wondering if there is something I could do on my end instead of waiting for the hosts to take action. Any advice? Also the log file states that

Replication finished with no explicit errors. Copying data from tmp tables to permanent

but looking at my BQ target, neither permanent nor tmp tables were created. Is there an additional issue with my BigQuery connection?

108b9406_6325_42fd_a35b_b1abde2f8ebf_logs_10_txt

Jana Kniel

03/21/2023, 10:01 AM

Hello everybody, i tried to connect LinkedIn Pages and as I try I always get the Error: ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):163 - {"serviceErrorCode":100,"message":"Not enough permissions to access: GET /organizations/****","status":403} I tried different organization pages and I am always a super admin in LinkedIn. Does somebody have the same problem and did find a solution?

jeremiah ishaya

03/21/2023, 10:39 AM

Hello Airbyte Community, Please have been working on a connector for some time using the low-code approach. However, when ever i try the spec command, the following error keeps popping up. Any assistance please? ERROR BELOW

Vibhor Agrawal

03/21/2023, 11:06 AM

@here We are evaluating airbyte as one of the option for connector solutions. I have question related to load data support related information with open source services? how much load data can be supported? also how this is different with other enterprise solution with airbyte. I am sorry if this information is already on documentation, as I was not able to find. Thanks in advance !! : )

Chetan M

03/21/2023, 11:10 AM

Failed to load file///home/chetanm/Documents FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last): File "/airbyte/integration_code/source_file/source.py", line 111, in check with client.reader.open(): File "/airbyte/integration_code/source_file/client.py", line 92, in open self._file = self._open() File "/airbyte/integration_code/source_file/client.py", line 139, in _open return smart_open.open(self.full_url, **self.args) File "/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 177, in open fobj = _shortcut_open( File "/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 363, in _shortcut_open return _builtin_open(local_path, mode, buffering=buffering, **open_kwargs) FileNotFoundError: [Errno 2] No such file or directory: '/home/chetanm/Documents i am getting above error when i give source as csv what path should i give in URL

Thiago Villani

03/21/2023, 11:55 AM

Hello, I'm configuring source MSSQL with CDC, ce destination postgresql, configuration according to documentation, but when doing the sync, the data is incrementing, with each sync, the amount of data doubles. documentation: https://docs.airbyte.com/integrations/sources/mssql Do you know what might be missing, or done wrong?

Arun RP

03/21/2023, 12:11 PM

Hey Everyone, Hope all doing good, We're planning to move and maintain a stream between rds postgres db to snowflake through airbyte, while moving the tables our tables are added with extra columns due to normalization, Is that any way that we can avoid that coz we're moving tables in terrabytes, if this extra columns are added it will go for more storage.. suggest me a way to do it with out that columns. my original table:

Copy code

Table "airbyte.movies"
 Column |          Type           | Collation | Nullable |                  Default                   
--------+-------------------------+-----------+----------+--------------------------------------------
 id     | integer                 |           | not null | nextval('airbyte.movies_id_seq'::regclass)
 src    | character varying(255)  |           |          | 
 name   | character varying(255)  |           |          | 
 genre  | character varying(255)  |           |          | 
 plot   | character varying(3000) |           |          | 
 rating | numeric                 |           |          | 
 year   | integer                 |           |          | 
 star   | character varying(500)  |           |          | 
Indexes:
    "movies_pkey" PRIMARY KEY, btree (id)

after moved it to snowflake:

Copy code

create or replace TABLE DEMO.MOVIES cluster by (_AIRBYTE_EMITTED_AT)(
	ID NUMBER(38,0),
	SRC VARCHAR(16777216),
	NAME VARCHAR(16777216),
	PLOT VARCHAR(16777216),
	STAR VARCHAR(16777216),
	YEAR NUMBER(38,0),
	GENRE VARCHAR(16777216),
	RATING FLOAT,
	_AIRBYTE_AB_ID VARCHAR(16777216),
	_AIRBYTE_EMITTED_AT TIMESTAMP_TZ(9),
	_AIRBYTE_NORMALIZED_AT TIMESTAMP_TZ(9),
	_AIRBYTE_MOVIES_HASHID VARCHAR(32)
);

Nikhil Sharma

03/21/2023, 2:35 PM

Hello, I just came here because I wasn’t able to click and open any of the tutorials on this page. https://airbyte.com/tutorials It doesn’t work on Chrome or Firefox. I see it works in Safari (maybe you all use Macs 🙂 )

Huib

03/21/2023, 2:49 PM

My airbyte instance is running on k8s, and for some reason, after a few days of running it “forgets” it should set resources requests+limits on pods - we have policies in place so that if don’t set them, scheduling is denied. Restarting the workers magically fixes the problem (jobs get scheduled just fine again), but the problem comes back a few days later. Any ideas?

Artem Kuzin

03/21/2023, 3:06 PM

Hello, could someone help me? Using 0.42.0v of Airbyte with MSSQL source and BigQuery destination. CDC turned on, and tests are passed while adding new sources. But there is no ‘incremental’ option in sync mode, only ‘full refresh’. Version of SQL driver is 1.0.4

Corbin Beebe

03/21/2023, 3:49 PM

Hello! We are running into some issues with the normalization process failing. The error thrown by our airbyte instance says it is due to the normalization process is not terminating normally. Any ideas what could be going on here?

Yuva

03/21/2023, 5:01 PM

Hi all, regarding custom transformations with dbt as here, I would like to understand if the “T” step / custom transformation happens in memory before the data is loaded to the destination? Or does it only apply once the source data is written to the destination (After “EL”) ?

brenda sofia

03/21/2023, 6:20 PM

Hello!! Can someone help me? I am trying to use S3 connector from Airbyte to send data from AWS S3 (csv files) to my Postgres database. When setting up and testing the connector everything seems ok, but the problem comes when trying to add a destination. When I click on add destination I get this message in the UI:

Copy code

Failed to fetch schema. Please try again

And in the logs I see this:

Copy code

File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 260, in _get_master_schema
    this_schema = schemas[file_info]

I took a deeper look about how the csv files look like and I found 2 issues: • " , " is the file delimiter and most of the lines in my csv file end with a comma • At the end of the file I have other values that are not required so I would like to ignore them since they don't follow a table-like schema Here is how the csv files look like:

Copy code

col1,col2,col3
val1,val2,val3,
val1,val2,val3,
val1,val2,val3,
val1,val2,val3,

More:
someVal:, $ 1234.12 USD
otherval:, $ 1234.12 USD
lastVal:, $ 1234.12 USD

What I would like to do it's to "perform" some pre-processing steps on my file before Airbyte loads the data. In this case I need to ignore

number of lines starting from the bottom of the file, as well as delete all the commas at the end of each line. Is there any way to do that? In addition, here is the file format configuration:

Slackbot

03/21/2023, 6:31 PM

This message was deleted.

Krisjan Oldekamp

03/21/2023, 6:42 PM

Running Airbyte helm chart (version 45.0) on GKE Autopilot. However, the

airbyte-server

is currently stuck in a CrashLoopBackoff. This error below keeps popping up: • ERROR Unable to load mask data from 'seed/specs_secrets_mask.yaml': null. Any idea what the issue could be? I was able to run this version successfully (also chart version 45.0), but after removing and reinstating the helm chart, it doesn't start up. I already tried to delete all the databases / fresh install but this didn't help.

Jordan Fox

03/21/2023, 6:55 PM

Getting an error booting a fresh install on an ubuntu linux box latest using version 0.42.0. Used the 'deploy airbyte on azure' guide, updated docker to newest version compatible with 0.42.0:

Copy code

airbyte-server                    | 2023-03-21 18:50:32 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Bean definition [io.airbyte.server.apis.SourceDefinitionApiController] could not be loaded: Error instantiating bean of type  [io.airbyte.commons.temporal.TemporalClient]
airbyte-server                    |
airbyte-server                    | Message: Unrecognized field "serve" (class io.airbyte.featureflag.ConfigFileFlag), not marked as ignorable (2 known properties: "enabled", "name"])
airbyte-server                    |  at [Source: (File); line: 4, column: 3] (through reference chain: io.airbyte.featureflag.ConfigFileFlags["flags"]->java.util.ArrayList[0]->io.airbyte.featureflag.ConfigFileFlag["serve"])
airbyte-server                    | Path Taken: new SourceDefinitionApiController(SourceDefinitionsHandler sourceDefinitionsHandler) --> new SourceDefinitionApiController([SourceDefinitionsHandler sourceDefinitionsHandler]) --> new SourceDefinitionsHandler(ConfigRepository configRepository,Supplier uuidSupplier,[SynchronousSchedulerClient schedulerSynchronousClient],AirbyteRemoteOssCatalog remoteOssCatalog,SourceHandler sourceHandler,AirbyteProtocolVersionRange protocolVersionRange) --> SynchronousSchedulerClient.synchronousSchedulerClient([TemporalClient temporalClient],JobTracker jobTracker,JobErrorReporter jobErrorReporter,OAuthConfigSupplier oAuthConfigSupplier,RouterService routerService,ConfigInjector configInjector) --> new TemporalClient(Path workspaceRoot,WorkflowClient client,WorkflowServiceStubs service,StreamResetPersistence streamResetPersistence,ConnectionManagerUtils connectionManagerUtils,NotificationUtils notificationUtils,StreamResetRecordsHelper streamResetRecordsHelper,[FeatureFlagClient featureFlagClient])
airbyte-server                    | io.micronaut.context.exceptions.BeanInstantiationException: Bean definition [io.airbyte.server.apis.SourceDefinitionApiController] could not be loaded: Error instantiating bean of type  [io.airbyte.commons.temporal.TemporalClient]
airbyte-server                    |
airbyte-server                    | Message: Unrecognized field "serve" (class io.airbyte.featureflag.ConfigFileFlag), not marked as ignorable (2 known properties: "enabled", "name"])
airbyte-server                    |  at [Source: (File); line: 4, column: 3] (through reference chain: io.airbyte.featureflag.ConfigFileFlags["flags"]->java.util.ArrayList[0]->io.airbyte.featureflag.ConfigFileFlag["serve"])
airbyte-server                    | Path Taken: new SourceDefinitionApiController(SourceDefinitionsHandler sourceDefinitionsHandler) --> new SourceDefinitionApiController([SourceDefinitionsHandler sourceDefinitionsHandler]) --> new SourceDefinitionsHandler(ConfigRepository configRepository,Supplier uuidSupplier,[SynchronousSchedulerClient schedulerSynchronousClient],AirbyteRemoteOssCatalog remoteOssCatalog,SourceHandler sourceHandler,AirbyteProtocolVersionRange protocolVersionRange) --> SynchronousSchedulerClient.synchronousSchedulerClient([TemporalClient temporalClient],JobTracker jobTracker,JobErrorReporter jobErrorReporter,OAuthConfigSupplier oAuthConfigSupplier,RouterService routerService,ConfigInjector configInjector) --> new TemporalClient(Path workspaceRoot,WorkflowClient client,WorkflowServiceStubs service,StreamResetPersistence streamResetPersistence,ConnectionManagerUtils connectionManagerUtils,NotificationUtils notificationUtils,StreamResetRecordsHelper streamResetRecordsHelper,[FeatureFlagClient featureFlagClient])
airbyte-server                    |     at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1921) ~[micronaut-inject-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:249) ~[micronaut-inject-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:3326) ~[micronaut-inject-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.context.DefaultBeanContext.finalizeConfiguration(DefaultBeanContext.java:3684) ~[micronaut-inject-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:341) ~[micronaut-inject-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:194) ~[micronaut-inject-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.runtime.Micronaut.start(Micronaut.java:75) ~[micronaut-context-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.8.6.jar:3.8.6]
airbyte-server                    |     at io.airbyte.server.Application.main(Application.java:15) ~[io.airbyte-airbyte-server-0.42.0.jar:?]
airbyte-server                    | Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.commons.temporal.TemporalClient]
airbyte-server                    |
airbyte-server                    | Message: Unrecognized field "serve" (class io.airbyte.featureflag.ConfigFileFlag), not marked as ignorable (2 known properties: "enabled", "name"])
airbyte-server                    |  at [Source: (File); line: 4, column: 3] (through reference chain: io.airbyte.featureflag.ConfigFileFlags["flags"]->java.util.ArrayList[0]->io.airbyte.featureflag.ConfigFileFlag["serve"])
airbyte-server                    | Path Taken: new SourceDefinitionApiController(SourceDefinitionsHandler sourceDefinitionsHandler) --> new SourceDefinitionApiController([SourceDefinitionsHandler sourceDefinitionsHandler]) --> new SourceDefinitionsHandler(ConfigRepository configRepository,Supplier uuidSupplier,[SynchronousSchedulerClient schedulerSynchronousClient],AirbyteRemoteOssCatalog remoteOssCatalog,SourceHandler sourceHandler,AirbyteProtocolVersionRange protocolVersionRange) --> SynchronousSchedulerClient.synchronousSchedulerClient([TemporalClient temporalClient],JobTracker jobTracker,JobErrorReporter jobErrorReporter,OAuthConfigSupplier oAuthConfigSupplier,RouterService routerService,ConfigInjector configInjector) --> new TemporalClient(Path workspaceRoot,WorkflowClient client,WorkflowServiceStubs service,StreamResetPersistence streamResetPersistence,ConnectionManagerUtils connectionManagerUtils,NotificationUtils notificationUtils,StreamResetRecordsHelper streamResetRecordsHelper,[FeatureFlagClient featureFlagClient])
airbyte-server

Yuva

03/21/2023, 8:18 PM

Does Airbyte sync to Snowflake randomly change the order of columns? At least for me , I have observed this behaviour. Running Airbyte on Docker

Nataly Merezhuk (Airbyte)

03/21/2023, 8:25 PM

Friendly reminder that we've got a code of conduct! Please make sure you are familiar with it before posting 👋 https://docs.airbyte.com/project-overview/slack-code-of-conduct/

Adam Roderick

03/21/2023, 9:29 PM

Hi @Augustin Lafanechere (Airbyte) can you see what is missing from my PR? It's been sitting and I'm not sure what to do to move it forward. This is my first attempt contributing and I'm sure I'm missing something. Can you help? https://github.com/airbytehq/airbyte/pull/23948

Alex Schwartz

03/21/2023, 11:17 PM

Hey folks! We are looking to enable the Sentry integration on our workers. I see we can add the

SENTRY_DSN

here: https://docs.airbyte.com/operator-guides/configuring-airbyte/#worker. What I don't see listed is a way to set a Sentry Environment string. We have multiple environments within our infrastructure so we typically segment our Sentry integrations that way. Is there a way to that with the Airbyte Sentry integration?

Ethan Veres

03/21/2023, 11:51 PM

Hi all! I’m trying to run locally and I’m getting this

Copy code

Unable to create SQL database.  {"error": "pq: database \"temporal\" already exists", "logging-call-at": "handler.go:97"}
airbyte-temporal                  | 2023-03-21T23:07:30.350Z    INFO    Starting schema setup   {"config": {"SchemaFilePath":"","InitialVersion":"0.0","Overwrite":false,"DisableVersioning":false}, "logging-call-at": "setuptask.go:57"}

I deleted all my local airbyte docker images and reran the script but still no go.

yuan sun

03/22/2023, 2:38 AM

Copy code

Hi,when compiling airbyte, the error is as follows:Build file '/Users/sunyuan/data/airbyte/build.gradle' line: 197

A problem occurred evaluating root project 'airbyte'.
> java.lang.UnsupportedClassVersionError: com/bmuschko/gradle/docker/DockerRemoteApiPlugin has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

* Try:
> Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.GradleScriptException: A problem occurred evaluating root project 'airbyte'.
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:93)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:133)
	at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:79)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:136)
	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.configuration.BuildOperationScriptPlugin.lambda$apply$0(BuildOperationScriptPlugin.java:62)
	at org.gradle.configuration.internal.DefaultUserCodeApplicationContext.apply(DefaultUserCodeApplicationContext.java:44)
	at org.gradle.configuration.BuildOperationScriptPlugin.apply(BuildOperationScriptPlugin.java:62)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$applyToMutableState$0(DefaultProjectStateRegistry.java:360)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:378)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.applyToMutableState(DefaultProjectStateRegistry.java:359)
	at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:42)
	at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
	at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:35)
	at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.lambda$run$0(LifecycleProjectEvaluator.java:109)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$applyToMutableState$0(DefaultProjectStateRegistry.java:360)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$fromMutableState$1(DefaultProjectStateRegistry.java:383)
	at org.gradle.internal.work.DefaultWorkerLeaseService.withReplacedLocks(DefaultWorkerLeaseService.java:345)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:383)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.applyToMutableState(DefaultProjectStateRegistry.java:359)
	at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.run(LifecycleProjectEvaluator.java:100)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:72)
	at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:762)
	at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:153)
	at org.gradle.api.internal.project.ProjectLifecycleController.lambda$ensureSelfConfigured$1(ProjectLifecycleController.java:63)
	at org.gradle.internal.model.StateTransitionController.lambda$doTransition$12(StateTransitionController.java:236)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:235)
	at org.gradle.internal.model.StateTransitionController.lambda$maybeTransitionIfNotCurrentlyTransitioning$9(StateTransitionController.java:196)
	at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
	at org.gradle.internal.model.StateTransitionController.maybeTransitionIfNotCurrentlyTransitioning(StateTransitionController.java:192)
	at org.gradle.api.internal.project.ProjectLifecycleController.ensureSelfConfigured(ProjectLifecycleController.java:63)
	at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.ensureConfigured(DefaultProjectStateRegistry.java:334)
	at org.gradle.execution.TaskPathProjectEvaluator.configure(TaskPathProjectEvaluator.java:33)
	at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:47)
	at org.gradle.configuration.DefaultProjectsPreparer.prepareProjects(DefaultProjectsPreparer.java:50)
	at org.gradle.configuration.BuildTreePreparingProjectsPreparer.prepareProjects(BuildTreePreparingProjectsPreparer.java:64)
	at org.gradle.configuration.BuildOperationFiringProjectsPreparer$ConfigureBuild.run(BuildOperationFiringProjectsPreparer.java:52)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
	at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
	at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
	at org.gradle.configuration.BuildOperationFiringProjectsPreparer.prepareProjects(BuildOperationFiringProjectsPreparer.java:40)
	at org.gradle.initialization.VintageBuildModelController.lambda$prepareProjects$2(VintageBuildModelController.java:84)
	at org.gradle.internal.model.StateTransitionController.lambda$doTransition$12(StateTransitionController.java:236)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
	at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:235)
	at org.gradle.internal.model.StateTransitionController.lambda$transitionIfNotPreviously$10(StateTransitionController.java:210)
	at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
	at org.gradle.internal.model.StateTransitionController.transitionIfNotPreviously(StateTransitionController.java:206)
	at org.gradle.initialization.VintageBuildModelController.prepareProjects(VintageBuildModelController.java:84)
	at org.gradle.initialization.VintageBuildModelController.getConfiguredModel(VintageBuildModelController.java:64)
	at org.gradle.internal.build.DefaultBuildLifecycleController.lambda$withProjectsConfigured$1(DefaultBuildLifecycleController.java:116)
	at org.gradle.internal.model.StateTransitionController.lambda$notInState$3(StateTransitionController.java:143)
	at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
	at org.gradle.internal.model.StateTransitionController.notInState(StateTransitionController.java:139)
	at org.gradle.internal.build.DefaultBuildLifecycleController.withProjectsConfigured(DefaultBuildLifecycleController.java:116)
	at org.gradle.internal.build.DefaultBuildToolingModelController.locateBuilderForTarget(DefaultBuildToolingModelController.java:57)
	at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.lambda$locateBuilderForTarget$0(DefaultBuildTreeModelCreator.java:73)
	at org.gradle.internal.build.DefaultBuildLifecycleController.withToolingModels(DefaultBuildLifecycleController.java:180)
	at org.gradle.internal.build.AbstractBuildState.withToolingModels(AbstractBuildState.java:123)
	at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.locateBuilderForTarget(DefaultBuildTreeModelCreator.java:73)
	at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.locateBuilderForDefaultTarget(DefaultBuildTreeModelCreator.java:68)
	at org.gradle.tooling.internal.provider.runner.DefaultBuildController.getTarget(DefaultBuildController.java:157)
	at org.gradle.tooling.internal.provider.runner.DefaultBuildController.getModel(DefaultBuildController.java:101)
	at org.gradle.tooling.internal.consumer.connection.ParameterAwareBuildControllerAdapter.getModel(ParameterAwareBuildControllerAdapter.java:39)
	at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.getModel(UnparameterizedBuildController.java:113)
	at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.getModel(NestedActionAwareBuildControllerAdapter.java:31)
	at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:97)
	at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
	at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:81)
	at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
	at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:66)
	at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
	at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:125)




how to solve

Mithilesh Kumar

03/22/2023, 4:40 AM

Hello everyone, We have deployed Airbyte on AWS Kubernetes using Helm and we have created mysql to redshift CDC connection, it was working fine but after some time getting this error

Copy code

2023-03-21 07:07:33 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):124 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:143) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:103) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:41) ~[io.airbyte-airbyte-commons-worker-0.41.0.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.41.0.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

also when testing connection we get this connection error for both source and destination. The connection issue getting resolved after restarting airbyte but again after sometime getting same issue. anyone can help here?

Sudhir Kenguva

03/22/2023, 6:15 AM

Hi Team, I am setting up sync from google sheet to postgres, every time the sync is happening, its giving access denied error. Any help please?