Hi Team, I have a question can we compile the github suorce code repo with docker ???

Hi All, In our syncs, found that Airbyte removes the milliseconds from the cdc_updated time

_ab_cdc_updated_at

This results in us having multiple records for the same ID with the same

_ab_cdc_updated_at

timestamp, while I think, having milliseconds would not give this problem. Is this something that can be configured? Source is Mysql

Hi, I opened a PR to fix the BambooHR source some time ago and I'm still waiting for a review https://github.com/airbytehq/airbyte/pull/23059 can someone help?

Hey everyone, we're currently considering moving from fivetran to airbyte cloud. Is there an info/stats on the costs? 🙂 • how much cheaper airbyte is? • did you consider airbyte + GBQ cost in total? will it increase or decrease due to the "incremental deduped history"? • any tips on how to migrate efficiently? I see that the fields name would change, since fivetran lowercases them by default • what do you think about the schema change implemented recently?

Hey everyone I try to connect with MySQL but I have this error “Access denied for user 'QA_DW'@'%' to database 'qa_dw'” I able to connect through workbench so, it should connect with Airbyte too. Can you help me?

Hello, I've asked a few times, but to no avail. I use airbyte having a MSSQL source and postgresql destination, from what I saw in the logs, the copy of the data in the origin is fast, but in the destination it takes time to do the normalization. Is there any good practice to improve performance on the destination, such as postgresql tuning?, or would it be the case of disk IO even though it can generate slowness? I use local airbyte with docker in an on-premise environment, with postgresql in a local environment as well. Thanks.

hello, curious as to a couple of things (please excuse my ignorant questions): 1. Could we use Airbyte to extract Kustomer data and insert certain data into Salesforce? 2. If yes to the above, what Airbyte product would be appropriate?

Hello, how can I set airbyte to fetch data from the source, only from a certain period?, using for example a where clause?, the way it is, it fetch all rows, I have tables with more than 15 years of data. Source MSSQL Destination Postgresql Thanks.

Hello, Is there a way to extract data from a field in

dynamodb

that has a restricted name? I am currently getting the following error, and I am wondering if there is a workaround without changing the source table column names?

software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved keyword: name

Thank you in advance!

@Marcos Marx (Airbyte) can you please tell me whether newly added

Auto Detect schema change

feature auto-populates any newly added column to destinations without needing to resync the whole Table? Or do we still need to resync the table again from scratch?

Hi Airbyte Team, I have submitted this PR for a New Source https://github.com/airbytehq/airbyte/pull/24138 . Can someone please help to review this and merge as soon as possible ? Its been almost 3 weeks now.

Message: Unrecognized field "serve" (class io.airbyte.featureflag.ConfigFileFlag), not marked as ignorable (2 known properties: "enabled", "name"]) Hi I am installing airbyte on my personal VM in gcp, How to resolve this?

Hi team, AIrbyte version: 0.40.6 Deployed on: K8s I am benchmarking airbyte w.r.t resources for achieving 50 parallel syncs. For that I have run some test cases to achieve 50parallel syncs. Based on that we can successfully run 50 parallel syncs this with 8vCPU and 16GB RAM. But the problem arises when we see the resource limits/requests applied on the sync pods which are spun up for syncing a connection. For one connection sync, we get these pods: • SOURCE-CHECK • SOURCE-READ • DESTINATION-CHECK • DESTINATION-WRITE By default, each of these pods contains 4 containers( 1. relay-stdout, 2. relay-stderr, 2. main, 3. init) and each container has the following resources limit/request quota configured. [ These are the default values]

resources:
      limits:
        cpu: 200m
        memory: 50Mi
      requests:
        cpu: 100m
        memory: 25Mi

At various points in the airbyte documentation, it is mentioned that we can tweak the sync pod size by changing the following environment variables in the deployment.

# Docker Resource Limits
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=

My question is quite straightforward, on what basis we can infer their values (for resource limits), as we do not have visibility on how much data we will be transferring over one connection sync (as it depends on the user, in our use case)?

Hi Team, I am trying to deploy the airbyte on docker containers. I am using docker compose file to deploy the services. But the process is getting stuck at the boot loader stage. I have tried to clean up an start the process from scratch but I am still getting the same results. Could anyone let me know why this is happening and how can I resolve this? I am using version 0.41.0 Deploying on a gcp vm instance. CC @King Ho

Hi team, When using Mongo as Source The basic Normalisation is not working. I have array with nested objects which is getting converted to string in destination database with is postgres.

Is there any way to trigger a manual sync via

octavia

or the

airbyte api

? I can’t seem to find it in the docs. Is this what I should use? https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/sync When I try this I get

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

Hello, I need help connecting Airbyte to an Oracle database using a wallet. Since I am using a wallet, I don't have the host IP address. Can you guide me on how to connect Airbyte to the database?

Hi Team, I am working on custom connector development, need one help. how i can use only connector code to build docker image instead of whole repo git@github.com:airbytehq/airbyte.git or i need to build custom connector having changes in in build connector. now i just want to use connector code only. is it possible ?

Hi there, Hope you’re well. I would like to know if anyone of you had already tried to extend or modify the

Salesforce Native connector

, so that we could write specific SOQL query and add filters ? Usecase : I need to use the convertCurrency() feature on some value columns to get amount converted from EUR -> USD. For instance, in a different tool we're trying to migrate from, we can type it that way:

SELECT *, 
convertcurrency(ACV) ACV_converted, 
convertcurrency(contract_value) contract_value_converted 
FROM opportunity

caveat: • We could ask our salesforce admin to add the converted column directly to the object itself, but I would like to keep that as a last resort • we do not want to pull 'report' from Salesforce since it's not scalable (we've already tried it in the past and we've maxed out its capacity) Thank you !

Hello team, We're using airbyte on a kubernetes cluster. It works well for all saas application we're using, however we just tried to perform some cdc replication from postgres (on rds) to snowflake (with pgoutput pluggin for logical decoding) and are experiencing some troubles. The first initial run successfully, but all the following fail with a

Producer failure java.lang.IndexOutOfBoundsException

-> I'll put the detailed logs in comments. Does any one already met a similar issue ? For information : we're using source-postgres:2.0.13 and destination-snowflake:0.4.59

Regarding using postgresql with wal2json plugin , i encountered errors and i see that the GUI does not provide the option. Up for visibility similar to this thread https://airbytehq.slack.com/archives/C021JANJ6TY/p1671837396118439

I have a question related to setting up Metrics for Datadog via Helm. I confused do I have to create a metrics pod or not? I’m trying my best to follow what I’m seeing here, but I’m getting way lost haha.

Hi ! Just wanted to know if there are any updates on when "Powered by Airbyte" will be released ? https://airbyte.com/embed-airbyte-connectors-with-api

Hi Guys

Do we have a GCS as a connector

?

Hey Guys I’m running the Incremental / Append update for FB Marketing and it creates duplicate rows for the insights report because of 28d lookback. Is there a primary key to deduplicate on?

Hey Guys! I`m running a PostgreSQL CDC to BigQuery and we are doing good with the backfill/first sync of 10 tables, but not in the next executions(sync). We have a sucess job, but with 0 records and no Errors or Warnings. Can you help understanding?

Can anyone provide direction on how to go about fixing this? Its a PGSQL -> MSSQL the column type is TEXT on PGSQL. There are null's in the column.. WARN i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$5):445 - Schema validation errors found for stream public.table. Error messages: [$.manager_email is of an incorrect type. Expected it to be string]

Hi all, just doing some evaluating of ingestion products and looking into AirByte. Need to be able to land the data in Databricks Unity Catalog, which the docs online suggest is doable but I've spun up the docker image the option isn't there, is it a cloud only option?