i need override mfa’s so i can continuously ingest data without mfa

i created a connection Source(postgres) --> Destination(S3) under transformations tab --> "Transformation and Normalization operations are not supported for this connection" is this expected behaviour?

HI all while writing logics in write function of destination, how tables can be created dynamically in destination ? Does airbyte provides any inbuilt function or class to create schema in destination dynamically for custom connectors. Thanks

Hi, I am doing a PoC for airbyte with mysql and redshift source. And wanted to know if I am creating a connection for first time and only want to sync partial data. Lets say data of 1 day and not whole table, is there a way to achieve this? Also lets say I run the connection manually can this be achieved to sync only partial data

Hello Everyone ! We have started exploring the airbyte where we have setup a connection as SQS-->AirByte-->S3. It works fine but the output from AirByte are stored as unreadable files. Afterwards, We downloaded these files from S3 and appended the files with ".gz" which helped to extract the file and view the contents inside it. I couldn't find any configuration which can be modified to fix it. Can someone please guide me if this is a default feature or am I doing something wrong here. Thanks in advance.

Hi everyone, I'm running a MS SQL Server (Full Refresh) job consisting of 12 tables and 12 views (4.8GB parquet size in S3). However this takes 1.5hours to complete and monitoring shows that the EC2 has more than 80GB of data going in and out. AWS DMS does the same job in a couple minutes. What could be the bottleneck here?

Hi Guys, i have this problem, https://github.com/airbytehq/airbyte/issues/12277 any help? Thanks

Hi Guys,

Hi guys, I have an Airbyte connection (mongoDB -> Postgres) with Normalisation as Raw Data and also a custom dbt transformation. I'm facing an issue where, whenever I use any cli args in custom transformation entry point Eg:

run --profiles-dir . --vars "{'dbname', 'postgres'}"

I end up getting this error "Could not find profile named 'normalize' ." always. However, if I change the entry point to just

run

, the entire sync and transformation work just fine. Please help

Hey team,

I am evaluating using airbyte for a new DW. For a POC i built a postgres to postgres connection for 5 tables local, docker deployment, basic normalization and incremental with dedup history replication. The initial load works great, but every subsequent sync fails where it trying to make a second jdbc connection back to the source db. Seems like a race condition, maybe due to both db instances on localhost w/ different ports ? This is all on a M1 mac so i had to build everything from latest to run native arm. Here is the error from log: 2022-04-26 134442 [43mdestination[0m > 2022-04-26 134442 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):126 - Preparing tmp table in destination started for stream django_celery_beat_crontabschedule. schema: public, tmp table name: _airbyte_tmp_khf_django_celery_beat_crontabschedule 2022-04-26 134442 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.PostgresSchema(printReplicaIdentityInfo):116 - REPLICA IDENTITY for 'public.django_celery_beat_periodictask' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns 2022-04-26 134442 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.PostgresSchema(printReplicaIdentityInfo):116 - REPLICA IDENTITY for 'public.django_celery_beat_crontabschedule' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns 2022-04-26 134442 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.PostgresSchema(printReplicaIdentityInfo):116 - REPLICA IDENTITY for 'public.django_celery_beat_intervalschedule' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns 2022-04-26 134442 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.PostgresSchema(printReplicaIdentityInfo):116 - REPLICA IDENTITY for 'public.django_celery_beat_clockedschedule' is 'DEFAULT'; UPDATE and DELETE events will contain previous values only for PK columns 2022-04-26 134442 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.PostgresStreamingChangeEventSource(searchWalPosition):268 - Searching for WAL resume position 2022-04-26 134442 [43mdestination[0m > 2022-04-26 134442 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):126 - Preparing tmp table in destination started for stream django_celery_beat_intervalschedule. schema: public, tmp table name: _airbyte_tmp_pdu_django_celery_beat_intervalschedule 2022-04-26 134442 [43mdestination[0m > 2022-04-26 134442 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):126 - Preparing tmp table in destination started for stream django_celery_beat_periodictask. schema: public, tmp table name: _airbyte_tmp_gth_django_celery_beat_periodictask 2022-04-26 134442 [43mdestination[0m > 2022-04-26 134442 [32mINFO[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):132 - Preparing tables in destination completed. 2022-04-26 134443 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.c.WalPositionLocator(resumeFromLsn):60 - First LSN 'LSN{0/241F10B8}' received 2022-04-26 134443 [44msource[0m > 2022-04-26 134442 [32mINFO[m i.d.c.p.PostgresStreamingChangeEventSource(searchWalPosition):287 - WAL resume position 'LSN{0/241F10B8}' discovered 2022-04-26 134443 [44msource[0m > 2022-04-26 134443 [32mINFO[m i.d.j.JdbcConnection(lambda$doClose$3):945 - Connection gracefully closed 2022-04-26 134443 [44msource[0m > 2022-04-26 134443 [1;31mERROR[m i.d.p.ErrorHandler(setProducerThrowable):31 - Producer failure 2022-04-26 134443 [44msource[0m > org.postgresql.util.PSQLException: FATAL: number of requested standby connections exceeds max_wal_senders (currently 1) 2022-04-26 134443 [44msource[0m > at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:613) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:161) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at org.postgresql.Driver.makeConnection(Driver.java:465) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at org.postgresql.Driver.connect(Driver.java:264) ~[postgresql-42.2.18.jar:42.2.18] 2022-04-26 134443 [44msource[0m > at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:230) ~[debezium-core-1.4.2.Final.jar:1.4.2.Final] 2022-04-26 134443 [44msource[0m > at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:871) ~[debezium-core-1.4.2.Final.jar:1.4.2.Final] 2022-04-26 134443 [44msource[0m > at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.reconnect(PostgresReplicationConnection.java:613) ~[debezium-connector-postgres-1.4.2.Final.jar:1.4.2.Final] 2022-04-26 134443 [44msource[0m > at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:142) ~[debezium-connector-postgres-1.4.2.Final.jar:1.4.2.Final] 2022-04-26 134443 [44msource[0m > at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:140) ~[debezium-core-1.4.2.Final.jar:1.4.2.Final] 2022-04-26 134443 [44msource[0m > at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113) ~[debezium-core-1.4.2.Final.jar:1.4.2.Final] 2022-04-26 134443 [44msource[0m > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?] 2022-04-26 134443 [44msource[0m > at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] 2022-04-26 134443 [44msource[0m > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] 2022-04-26 134443 [44msource[0m > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] 2022-04-26 134443 [44msource[0m > at java.lang.Thread.run(Thread.java:833) [?:?]

hi team, looking for an update on this when possible! https://airbytehq.slack.com/archives/C021JANJ6TY/p1650374439292259

hi

I came here for some help

For more details see Big Query Error collection: BigQueryError{reason=accessDenied, location=null, message=Access Denied: File gs://airbyte-vt/test/unified_d15-449f-9a3f-718e0ee5aa8e/1.avro: Access Denied}: 2022-04-26 214552 destination > 2022-04-26 214552 ERROR i.a.i.d.b.BigQueryGcsOperations(lambda$copyIntoTmpTableFromStage$0):141 - [JobId{project=pagos-281413, job=9c0fbf20-a1ee-4a20-8d3d-cdee07c1d2fd, location=US}] Failed to upload staging files to tmp table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=unified_model_fg, tableId=_airbyte_tmp_uby__databill}} (unified_model_fg) 2022-04-26 214552 destination > com.google.cloud.bigquery.BigQueryException: Error is happened during execution for job: Job{job=JobId{project=pagos-281413, job=9c0fbf20-a1ee-4a20-8d3d-cdee07c1d2fd, location=US}, status=JobStatus{state=RUNNING, error=null, executionErrors=null}, statistics=LoadStatistics{creationTime=1651009552397, endTime=null, startTime=1651009552548, numChildJobs=null, parentJobId=null, scriptStatistics=null, reservationUsage=null, inputBytes=null, inputFiles=null, outputBytes=null, outputRows=null, badRecords=null}, userEmail=, etag=fOUMPw0OR2YyWGJT8s4ozA==, generatedId=pagos-281413:US.9c0fbf20-a1ee-4a20-8d3d-cdee07c1d2fd, selfLink=, configuration=LoadJobConfiguration{type=LOAD, destinationTable=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=unified_model_fg, projectId=pagos-281413, tableId=_airbyte_tmp_uby__databill}}, decimalTargetTypes=null, destinationEncryptionConfiguration=null, createDisposition=null, writeDisposition=WRITE_APPEND, formatOptions=FormatOptions{format=AVRO}, nullMarker=null, maxBadRecords=null, schema=Schema{fields=[Field{name=_airbyte_ab_id, type=STRING, mode=null, description=null, policyTags=null}, Field{name=_airbyte_emitted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null}]}, ignoreUnknownValue=null, sourceUris=[gs://airbyte-vt/test/unified_model_fg__databill/2022/04/26/21/b1ab1fd9-3d15-449f-9a3f-718e0ee5aa8e/1.avro], schemaUpdateOptions=null, autodetect=null, timePartitioning=null, clustering=null, useAvroLogicalTypes=true, labels=null, jobTimeoutMs=null, rangePartitioning=null, hivePartitioningOptions=null}}, 2022-04-26 214552 destination > For more details see Big Query Error collection: BigQueryError{reason=accessDenied, location=null, message=Access Denied: File gs://airbyte-vt/test/unified_mo3d15-449f-9a3f-718e0ee5aa8e/1.avro: Access Denied}: 2022-04-26 214552 destination >

I am having an error for permissions, can someone tell me what I am missing y my HMAC configuration.

Is it possible to trigger a rerun of the default normalization Job without re-syncing the data?

Hello everyone...i am implementing creating source defintion api but i recieve the following error of 500.What could be the solution to this???

This message was deleted.

Hey.. I’m working on a change to a connector and I want to test it locally. I see in this README that I need to build the solution with

gradlew

but I’m getting an error when running

SUB_BUILD=PLATFORM ./gradlew build

Starting a Gradle Daemon (subsequent builds will be faster)

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the User Manual chapter on the daemon at <https://docs.gradle.org/7.4/userguide/gradle_daemon.html>
Process command line: /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java -Xss4m --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED -Xmx4g -Dfile.encoding=UTF-8 -Duser.country=NZ -Duser.language=en -Duser.variant -cp /Users/maikel.penz/.gradle/wrapper/dists/gradle-7.4-bin/c0gwcg53nkjbqw7r0h0umtfvt/gradle-7.4/lib/gradle-launcher-7.4.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 7.4
Please read the following process output to find out more:
-----------------------
Unrecognized option: --add-exports
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.


* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

I tried many things already and I think its something with my gradle installation. I’ve been working on a Python connector so I’m very new to using gradle + java

Hey guys, I want to launch all my Airbyte connections using a Python script and for that I'm trying to understand the Airbyte API. However, for all connection management related API requests in the documentation (https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/list), we must always pass the connexion_id to trigger the connection or even to retrieve its details. How can I retrieve my connexion_id after creating the connection manually? Thanks a lot and hope you guys have a great one !

Hey y'all, I am finding Airbyte to be extremely slow and at the end it doesn't work Is it me not using it correctly or is it something common?

Hi there, I have a question around cancelling Connection runs: I have tried, unsuccessfully, with both Airflow as well as Dagster to cancel running connection job. Terminating a job in Dagster or a task in Airflow has no effect whatsoever in Airbyte. The only thing which seems to be working as intended but is not implemented by either

dagster-airbyte

nor the

apache-airflow-providers-airbyte

is to cancel a job via the Cancel job API endpoint via

/v1/jobs/cancel

, which works as intended. How do I cancel Airbyte jobs without falling back on the API?

Hello there, I am trying to connect a Gsheet as a source (with SnowFlake as a destination). The header is not on the first but row no.3 of the sheet and once ingested, the value (i.e. column names) are ingested like any other row. I saw that we could add reader_options when the source was a file, any chance we can do something similar for gsheet? (I couldn’t see any other thread with the same issue, which is surprising). If not, what would be the alternative? storing the sheet as a .csv or .xls and import it from a S3 or GCS? Many thanks !

Hi all. I’m just getting started with Airbyte. Are the Redshift source and destination still in alpha and beta? I’m interested in using them for sure, but I’m wondering if I should wait until they’re in “Generally Available.” I’m looking at this doc https://docs.airbyte.com/integrations/

Hi, all does Airbyte work on Mac M1 machines? I am getting errors while running from docker. Is there any supporting document to tackle this.

Hi all. Is this a true or false statement? "The

WEBAPP_URL

environment variable is only used as text for job notifications."

Hi All, I just found Airbyte yesterday as I need to sync a MS SQL DB to AWS Dynamo DB. I have MSSQL connected no issue but can not get the connection to DynamoDB to work. Any pointers? I'm running the docker image locally just for testing purposes.

quick question - the

IncrementalMixin

can be used just as well from

parse_response()

as it can be from

read_records()

right? I was under the impression that

read_records()

shouldn't generally be overridden

Hi All, has anyone knows how to deploy Airbyte to Heroku? Is it possible? Will it work? Thank you 🙏