is there any way to know how many tables left for sync directly without logs ?

Is there a way to trigger a full refresh of an incremental-deduped table, without clearing the SCDs?

what is the logic for dynamic batch size in sync ?

how to increase sync speed in postgres to postgres replication

how sync works internally ? I am not asking about sync mode

what is InMemoryRecordBufferingStrategy and flushing

does dynamic batch size is reduced when ram provided to source-worker is less ?

how to set destination buffer size in postgres connector

Hey all, I would like to ask a question: Why is Airbyte’s connection check so slow when adding a source?

Hey all, we are running on the new declarative intercom connector (0.2.0) and with every sync we have a small number (~0,01%) of contacts that don’t show up in the destination. The total contact count in the sync logs is always lower than the total number of contacts on Intercom. Could someone help me make sense of the manifest.yaml and debug this? My current theory is that it is somehow linked to the pagination. We are running in a full refresh mode, and I see that the “contacts” declaration is under “stream_incremental_search”. • Why is it using the

contacts/search

endpoint instead of

contacts

? • Given that the results are sorted using the

updated_at

field, could it be that the pagination breaks because while fetching the records some of them will get updated and skip to the end of the list? • What is the purpose of the record filter condition? (

{{ record['updated_at'] >= ( stream_state.get('prior_state', {}).get('updated_at', 0) if stream_state else stream_slice.get('prior_state', {}).get('updated_at', 0) ) }}

)

Hi all, I use the Hubspot connector for engagements and it seems that the connector is not replicating all the data from engagement. We have 60k engagements in Hubspot and Airbyte push only 8k records in the table engagements. Did someone notice this ?

Hello, i am having t3-large ec2 instance with 8gb ram. after sometime my airbyte is not opening. i am getting like Internal Server Error: Error getting connection from data source HikariDataSource (HikariPool-1) i am having 4gb free ram and 20 gb of space. can you help to fix this issue

get webhook notification before every sync if source schema has changed , not just once in 24 hours

Hi Folks, I have a connection with s3 destination taking 90% of time of the execution cleaning the s3 folders. Is this your first time deploying Airbyte: No Memory / Disk: 8Gb / 16GB Deployment: EKS Airbyte Version: 0.41.0 Source name/version: Postgres 1.0.42 Destination name/version: S3 0.4.0 Step: Running the connection Description: The connection extracts data from 32 tables, 25 using full method. The connection is taking 6 hours and 20 mins to get done... It took 6h and 10m cleaning the S3 folders... and 9 minutes to extract data. *C*ould you folks suggest any tip or approach to have a better performance on this case?

🔥 Community Office Hours starts in 15 minutes 🔥 Q&A - No topic, ask anything! At 1pm PDT click here to join us on Zoom!

Hi all, when transferring Google Analytics 3 (Universal Analytics) data to BigQuery, is it possible to transfer events? This list of dimensions and metrics don't mention them. I also tried adding them as a custom report, following the naming convention, like

ga:eventCategory

, but no data was transferred.

can aws rds read replica of postgres be used as source database for log-based incremental cdc sync ?

how to manage schema changes when postgres cdc incremental is used for sync ?

Hi All, Has anyone had the issue of getting the error message `Server temporarily unavailable (http.502.iLso8sA45wmExBN8oxeZ2z)`when setting up an S3 bucket as a new source?

Hi, I'm trying to create connector from LinkedIn Ads, but I got this error:

Configuration check failed

HTTPError('403 Client Error: Forbidden for url: <https://api.linkedin.com/v2/adAccountsV2?count=1&q=search>')

I’m trying to use BigQuery as a destination (with Github as a source), and i’m not seeing the option to normalize the data. This results in only the

_raw_

tables being created in BigQuery. seeing this when I goto Transformations tab. This used to work before I upgraded Airbyte and the connector versions. Would appreciate any help!

Hello all 🙂 I got this error, any help will be appreciated :) Airbyte version: 0.44.5 MongoDb connector

PUT <https://www.googleapis.com/upload/bigquery/v2/projects/doyoogo-events/jobs?uploadType=resumable&upload_id=ADPycdvVkPvgaZUfvREhf9iJPn6RCWOMDYymB4ClIto_u9NvNUdI4qj77R2S8uN_Fws8Cecg2a5oeQMG0x53yhXFDgjw1ggXK_X8>
{
  "error": {
    "code": 400,
    "message": "Request was malformed.",
    "errors": [
      {
        "message": "Request was malformed.",
        "domain": "global",
        "reason": "badRequest"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

Hello, I have a question that's specific to Airbyte docs (not sure if this channel is the right place though). Can I view the Airbyte docs for an older release/version? Currently https://airbytehq.github.io/ shows documentation only for the current version Thanks in advance! 😃

Hi Folks, Regarding GDPR, PII and other requirements that may arise... Does Airbyte provide any approach to avoid extracting sensible columns from any source? is this subject being discussed or is it on the roadmap?

When there is a non-breaking schema change with an incremental-dedupe sync, if I reset the stream upon accepting the schema changes, do I lose my SCD history up to that point?

Issue with Google Analytics 4 connector As you can see in the screenshot, the data between the different streams is varying quite a lot. I would expect all streams to display the same amount of sessions and totalUsers. The data via the Airbyte connector is also nowhere close to the data that is taken directly from GA4 (Source of Truth). So at this moment it is impossible to have reliable data from the Google Analytics 4 connector, which makes it unusable. Or am I missing something? In advance, thanks for any help! FYI: I set the connection to sync every 12h and I can see in the raw data in BigQuery, that there are sometimes multiple entries for the same date (e.g. in the website_overview stream). I would expect the connector to overwrite the data for each day if there are multiple syncs per day (specifically for the website_overview stream), which is apparently not the case. • Airbyte Deployment via plural.sh • Airbyte Version: 0.44.2 • Source name/version: GA4 - 0.2.0 • Destination name/version: BigQuery - 1.3.4

Hi, I ran 'python -m pytest -p integration_tests.acceptance' but encountered three errors. The first error is related to the 'conftest.py' file located in 'airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test'. I believe this error occurred because the test is searching for a Docker image on Docker Hub, but we don't have any. My proposed solution is to modify line 149 to 'return image_tag'. The second error is related to reading two sequential readings. I think the error occurred because we assigned a unique identifier to each stream record. To address this, my solution is to set the ID to None for each record in the 'for' loop on lines 52 and 58 of 'airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py'. The third error is that the key 'Error' is missing in 'airbyte/airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/utils/connector_runner.py'. My proposed solution is to wrap 'Error' with '.get()', line 163. I have attached screenshots of the errors in the specified order. I hope you can assist me with these issues. Thank you.

Hi. I am have a problem. When I incremental data from mongodb to bigquery by airbyte. My createdAt field on mongodb is being detected by airbyte as string. That leads to the partition on my bigquery table not createdAt. It is ingest date. Is there any way to slove this problem. CreatedAt Filed is type Date on mongo. Thanks you

I have an issue where after a Reset of my Salesforce to Postgres connection not all records in Salesforce will be emitted for the given stream (despite other connections such as my Salesforce to BigQuery connection having all records). This doesn’t produce an error and the only related logs I can find refer to a possible state collision. However, this is present on both successful and unsuccessful syncs. Airbyte: 0.44.4, Salesforce: 2.0.13, Postgres: 0.3.27.

2023-05-22 18:32:09 WARN i.a.w.i.b.DefaultSyncStatsTracker(updateDestinationStateStats):195 - The message tracker encountered an issue that prevents committed record counts from being reliably computed. This only impacts metadata and does not indicate a problem with actual sync data.
io.airbyte.workers.internal.book_keeping.StateDeltaTracker$StateDeltaTrackerException: State hash 1727035794 was already committed, likely indicating a state hash collision
	at io.airbyte.workers.internal.book_keeping.StateDeltaTracker.commitStateHash(StateDeltaTracker.java:121) ~[io.airbyte-airbyte-commons-worker-0.44.4.jar:?]
	at io.airbyte.workers.internal.book_keeping.DefaultSyncStatsTracker.updateDestinationStateStats(DefaultSyncStatsTracker.java:192) ~[io.airbyte-airbyte-commons-worker-0.44.4.jar:?]
	at io.airbyte.workers.internal.sync_persistence.SyncPersistenceImpl.updateDestinationStateStats(SyncPersistenceImpl.java:447) ~[io.airbyte-airbyte-commons-worker-0.44.4.jar:?]
	at io.airbyte.workers.internal.book_keeping.AirbyteMessageTracker.handleDestinationEmittedState(AirbyteMessageTracker.java:137) ~[io.airbyte-airbyte-commons-worker-0.44.4.jar:?]
	at io.airbyte.workers.internal.book_keeping.AirbyteMessageTracker.acceptFromDestination(AirbyteMessageTracker.java:103) ~[io.airbyte-airbyte-commons-worker-0.44.4.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromDstRunnable$4(DefaultReplicationWorker.java:324) ~[io.airbyte-airbyte-commons-worker-0.44.4.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

🔥 Community Office Hours starts in 15 minutes 🔥 Deep Dive Sessions: airbyte-bootloader At 1pm PDT click here to join us on Zoom!