Hello comunity, I have a issue with error "Additional Failure Information: scheduledEventId=48, startedEventId=49, activityType='RunWithJobOutput', activityId='5e061e7a-b6fe-36ec-b547-245364404b28', identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED" How can i resolved this issue?

Hi everyone! We are having issues to replicate data from a ClickHouse source to a ClickHouse destination. Regardless of the amount of memory the Clickhouse instance(dest) has, the sync process always hangs with “Memory limit (total) exceeded” error. And this happens during the normalization step:

2023-01-26 14:57:07 normalization > 6 of 9 ERROR creating sql incremental model postgres.user_chapters_scd ................................................. [ERROR in 28.23s]

2023-01-26 14:57:07 normalization > Database Error in model user_chapters_scd (models/generated/airbyte_incremental/scd/postgres/user_chapters_scd.sql)
2023-01-26 14:57:07 normalization >   :HTTPDriver for <http://ip-10-0-11-80.ec2.internal:8123> returned response code 500)
2023-01-26 14:57:07 normalization >    Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.82 GiB (attempt to allocate chunk of 4259816 bytes), maximum: 27.69 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (avg_value_size_hin
2023-01-26 14:57:07 normalization > Done. PASS=7 WARN=0 ERROR=1 SKIP=1 TOTAL=9
2023-01-26 14:57:07 normalization > Code: 241. DB::Exception: Memory limit (total) exceeded: would use 27.82 GiB (attempt to allocate chunk of 4259816 bytes), maximum: 27.69 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (avg_value_size_hint = 311.873046875, avg_chars_size = 364.64765625, limit = 8192): (while reading column _airbyte_data): (while reading from part /var/lib/clickhouse/store/1b7/1b7bc2ee-08fa-4d83-a701-ae65cbb607a1/all_35_66_2/ from mark 440 with max_rows_to_read = 8192): While executing MergeTreeThread. (MEMORY_LIMIT_EXCEEDED) (version 22.12.3.5 (official build))
2023-01-26 14:57:07 normalization >

Can anyone help us please? 🙏

validate table chapter_sdc_sql

Hi everyone, I encounter an issue with locally run airbyte, more precisely with with database which constantly restarts. I can not go inside to see what happens, however ,as I understand, if put

docker compose down

then all information that stored there will be lost. I know that there is json file where additional info is stored, what is the path to that? How to save the data in more reliable way, do you deploy a db separately not in docker or you can somehow mount a disk and dump copies there? What are best practices in general administrating local airbyte instance? Thank you in advance 🙂

hello everyone i'm facing issue in normalization phase and this is the log

Is there any way to investigate where a bottleneck in transfer is? Like if it’s the Source or Destination limiting speeds?

Also for public connectors, like the shopify one, where is the normalization dbt models kept in github?

Hi everyone, which channel should I go to if I want to request a new stream to be added to airbyte's existing google ads connector?

I'm trying to add customize GAQL in google ads connector to create new report that airbyte doesn't provide currently, this is the GAQL:

SELECT segments.keyword.info.match_type, segments.keyword.ad_group_criterion, segments.keyword.info.text FROM search_term_view

but after syncing, I don't see any information related to this customized query that I specified, am I missing something? Attached is my log file, TIA

Running mongo version 3.0.14, when trying to do an initial sync, I get:

2023-01-26 20:44:01 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}
2023-01-26 20:44:01 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-26 20:44:01 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-26 20:44:01 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Cluster created with settings {hosts=[10.0.4.144:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2023-01-26 20:44:01 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Cluster description not yet available. Waiting for 30000 ms before timing out
2023-01-26 20:44:02 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Opened connection [connectionId{localValue:2, serverValue:264603}] to 10.0.4.144:27017
2023-01-26 20:44:02 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Monitor thread successfully connected to server with description ServerDescription{address=10.0.4.144:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=3, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=22736978, setName='privacy_test', canonicalAddress=10.0.4.144:27017, hosts=[10.0.4.144:27017, ip-10-0-4-226:27017], passives=[10.0.4.147:27017], arbiters=[], primary='10.0.4.144:27017', tagSet=TagSet{[]}, electionId=639a5356a91e3408be5c4441, setVersion=5, topologyVersion=null, lastWriteDate=null, lastUpdateTimeNanos=2221618429251}
2023-01-26 20:44:02 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Opened connection [connectionId{localValue:1, serverValue:264604}] to 10.0.4.144:27017
2023-01-26 20:44:02 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Opened connection [connectionId{localValue:3, serverValue:264607}] to 10.0.4.144:27017
2023-01-26 20:44:02 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):116 - Something went wrong in the connector. See the logs for more details.
Stack Trace: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: ReadConcern not supported by wire version: 3

Hello I am trying to run the latest version of the docker-compose file and I keep running into an issue with the temporal namespace being set. The temporal container says that it was already registered, but the rest of the containers just keep looping with "Waiting for namespace default to be initialized in temporal...". I have tried this on another machine with an older version of the docker compose that I have successfully run in the past (Alpha-0.36.2) and it also is running into an issue with the temporal namespace

Hi, just wondering if there is a native way to assign Read permissions on the destination normalised table (Postgres, in my case) after the sync is complete. Currently, I'm solving this by running a shell script after the sync is complete. It would make much more sense if there was a step in the AirByte replication connection that would allow you to configure this.

Hello everyone! I am getting this message: Additional Failure Information: When searching for a relation, dbt found an approximate match. Instead of guessing which relation to use, dbt will move on. Please delete "CIMED"."PUBLIC"."order", or rename it to be less ambiguous Someone can help me?

I noticed keyword names are missing from the BingAds connector and we only see keyword_id and keyword_status. Is this a known gap, perhaps because of the limitation on the basic streams being loaded in currently?

hi. Airbyte team, Need help with review for this PR https://github.com/airbytehq/airbyte/pull/21242

Hi, I wanna ask about incremental+dedup loading case. I have upgraded Airbyte from 0.39-alpha to 0.40.28 for log retention feature. This case is started when using the newest version. Sync incremental+dedup always scan/emit huge data size (almost same with reset-sync operation or full-refresh mode) every running from source (Postgres). The destination (BigQuery) is seems good (contains expected data). Sadly, this big load made our VM’s CPU is on spike with more time-consuming process every time. So far, I found that huge emitting data case happens on multi tables/streams (in the same job/connection). Otherwise, single table/stream (in one job/connection) has expected pattern (smaller number of data emittance). *NB, I have tried refreshing schema, resetting, deleting all staging table manually, but the multi-stream connection still emit/scan huge data on incremental+dedup mode. The single stream connection is working as expected probaly because there is schema change in the source when resetting or refreshing the connection.

Shopify connector ran for 21h, synced 1.6m records and lost connection to the destination for a few minutes. And on next sync starts all over again 😞 Is that expected? The sync model is incremental + dedeup history

Hello everyone 🙂 Can someone help me in determining what was the cause of failed acceptance test in my PR? I'm trying to fix the indefinitely looping in Okta connector's GroupMembers stream but I cannot understand why was the users stream failed even though I've only modified the GroupMembers stream? https://github.com/airbytehq/airbyte/pull/20833

Hey guys, I'm currently working on a schema, which has an array of objects. When I run the Basic Normalization, it doesn't create new tables for the objects inside the array, although I specified the objects. So does anyone of you has an idea of how a schema should look like?

hey guys, do any of you use Airbyte with podman-compose?

Hello everyone, I have deployed Airbyte locally. I have no idea where I should copy my ssl certificate for using replication from Postgresql to Clickhouse. Maybe it has any tutorials?

Hi All I'm trying to make a connection to MongoDB , this is the following error sun.security.provider.certpath.SunCertPathBuilderException unable to find valid certification path to requested target}, caused by {sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}, caused by {sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}}] I installed InstallCert.java in the docker image of the airbyte server following this https://github.com/escline/InstallCert Still my server was not able to connect to MongoDB, can someone please help @all

Hello comunity, I have a issue with error "Additional Failure Information: scheduledEventId=48, startedEventId=49, activityType='RunWithJobOutput', activityId='5e061e7a-b6fe-36ec-b547-245364404b28', identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED" How can i resolved this issue? GKE with 16+nodes and 29gb. Actually have 200 connections schedule 00 AM bd is as400 connector jdbc load method is: GCS worker configurations: - name: MAX_SYNC_WORKERS value: "400" - name: MAX_SPEC_WORKERS value: "200" - name: MAX_CHECK_WORKERS value: "200" - name: MAX_DISCOVER_WORKERS value: "200" - name: SYNC_JOB_MAX_ATTEMPTS value: "10" - name: SYNC_JOB_MAX_TIMEOUT_DAYS airbyte connectos are fail value: "3"

Hey everyone, I have a question related to temporal storage in airbyte. When syncs are run, the data is read from source and written on the destination, i*n transitioning of data, where does it store the data temporarily?* If it stores in the cache, how to control the data persistence time, (like after 123 mins, clear the cache)?? And at which directory/path the data is stored.? I'd be highly grateful if anyone can help me understand this, as data caching time is very important for us to manage great workload. cc: @Grace Stern (Airbyte)

Hey there, do you have any kind of partner integration that would enable a SaaS to provide 1-click Airbyte Cloud account creation for their users?

Hi all, I wanted to ask if you experienced issue with airbyte-db

2023-01-26 17:13:02.732 UTC [1] FATAL:  could not write lock file "postmaster.pid": No space left on device

and what is the way to increase the size? I added memory to my server, so it shoudn’t be the issue, but I need somehow to resize the size of airbyte-db image … Thanks in advance!

exist documetation to ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS:30 ACTIVITY_MAX_ATTEMPT:5 ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS:600 ACTIVITY_MAX_TIMEOUT_SECOND:120

Hi all, I'm trying to install airbyte on EKS - fargate. (I didn't mount a file system to fargate like EFS) Database is Aurora postgres. I've disabled minio and used S3. But server pod doesn't start properly. Any advice?

Hello Team, I received notification 'Non-breaking schema updates detected' for my connection S3->Postgres, when I click review changes, I received error 'Failed to fetch schema. Please try again Error: non-json response'. And I'm not capable to change any settings about this connection without finishing review the schema update, so I'm kind of block. Any solutions/idea about this problem? Thx

Hi everyone, I wanna ask about Dagster+Airbyte combo. We have configured those two and it has been operating until now. But we often receive “Exceeded max number of retries” in Dagster-Airbyte job (I am sure its called asset materialization). So we can’t fully depends on Dagster trigger/sensor mechanism based on Airbyte asset materialization in Dagster (context, we want to run some Dagster operation/job after certain Airbyte asset has been updated). So, is there any workaround for this thing? Configuration & Specs Dagster 1.0.9 Airbyte 0.40.28 request_max_retriex : 15 (Dagster-Airbyte Asset) request_retry_delay : 2 (Dagster-Airbyte Asset) We use default value for all Airbyte limit env