Please is there a specific channel for help on developing sources and Destinations?

Hi all, I am new to Airbyte, I just signed in for Airbyte cloud free trial. My usecase is I wanted to sync my data from Lago to Redshift. I can see in the documentation that airbyte provide support for Lago connector but when I am trying to setup my first connector using Lago as Source, Lago is not appearing in the available sources. Can someone help me understand if Lago is supported as a source as of now or not? cc: @User

Hi team 👋 I’m trying to update Airbyte from 0.40.14 to 0.40.26, and the booloader fails because it tries to upgrade my custom connectors, which don’t follow semantic versioning for tag names. I know this issue already happened before, and was solved with this PR, and I believe the bug appeared again after this PR. Should I open an issue for this? Bootloader logs

...
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(runFlywayMigration):398 - Migrating jobs database
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully validated 11 migrations (execution time 00:00.007s)
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Current version of schema "public": 0.40.18.002
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema "public" is up to date. No migration necessary.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):216 - Ran Flyway migrations.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(createWorkspaceIfNoneExists):317 - workspace already exists for the deployment.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):219 - Default workspace created.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(createDeploymentIfNoneExists):307 - running deployment: 5c1e816f-dada-4e28-bcf2-fd430ae877c1
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):222 - Default deployment created.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):225 - Set version to AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'}
2022-12-30 11:44:28 INFO i.a.c.p.ActorDefinitionMigrator(updateConfigsFromSeed):72 - Updating connector definitions from the seed if necessary...
2022-12-30 11:44:28 INFO i.a.c.p.ActorDefinitionMigrator(updateConfigsFromSeed):75 - Connectors in use: [airbyte/destination-snowflake, <http://464622532012.dkr.ecr.us-east-1.amazonaws.com/dd-airbyte|464622532012.dkr.ecr.us-east-1.amazonaws.com/dd-airbyte>]
Exception in thread "main" java.lang.IllegalArgumentException: Invalid version string: source-s3-0.1.18
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
	at io.airbyte.commons.version.Version.<init>(Version.java:37)
...

I’m trying to deploy on k8s

Is there a reason why

UserLocationPerformanceReport

is not included in the Bing Ads connector? I can get that through Fivetran (it's pretty large), but also Meltano doesn't support it.

hi lovely people kindly help me setup my collecting data for companies and im in south africa i welcome all help

Hello - We have an issue with an AirByte pipeline. Our current setup: • AirByte deployed on a VM on GCP • AirByte talks to a SQL Server VM in Azure • CDC is enabled on a bunch of tables in SQL Server VM • Pipeline writes data to a BQ dataset in incremental + dedup mode The Problem: • A couple of tables have data mismatch • These tables have FLOAT, Char and varchar columns, and the UPDATES are not propagating to the target tables • The missing data is also not found on SCD tables or even _tmp tables as well. • No Errors in the sync logs found Is this a known issue with the SQL Server connector?

the problem: The sync works fine for ALL tables, except for this busy transactions table that gets updated multiple times throughout the day.

there is data mismatch between the source and the destination, blank columns

is there any known issue with SQL Server CDC and AirByte for busy tables?

Hello all, can someone let me know, how to import file from s3 bucket to snowflake with no header. I need to pass schema externally and all the data types are string

Hi guys, i am deploying customised connector - works fine - but i need to make it public 🙂

so it can be visible in all workspaces

how can i do that

??

State code: 08S01; Message: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

I am using Ubuntu 20.04, trying to create mysql source connector, and I have been getting the error above. I don't know what to do to get the issue solved. Help will be very appreciated

Hi all,

Hi all, I'm trying to get deeper into the Incremental Append streams. Sometimes I see a lot of messages like these in case of first sync logs: 2022-12-21 102706 source > State report for stream campaigns_email_content - original: null = null (count 0) -> latest: id_email = 17022278 (count 1) I suppose it means creation of "incremental cursor" of some kind, am I right? The strange thing is - in some syncs there is a lot of them, in another syncs there are almost none, just the "data pump" messages: ... 2022-12-21 092103 destination > Flushing email_content: 65 records (24 MB) 2022-12-21 092103 source > Reading stream email_content. Records read: 750000 2022-12-21 092103 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):347 - Records read: 750000 (55 GB) 2022-12-21 092103 destination > Flushing email_content: 94 records (24 MB) 2022-12-21 092104 destination > Flushing email_content: 96 records (24 MB) ... My questions are - is it an indication of cursor creating or what? Why does it take so much time and can I add resource to make it go faster? And what happens if I reset the data, is the cursor deleted as well?

Hi @Engineering Team I am new to airbyte Facing problem with Azure Dedicated SQLPool default schema creation "enforced unique constraints are not supported." as per MS DOC, we need to use Syntax like below: could you please check? @Timam

Hi @Engineering Team how to set timeout jobs in Airbyte opensource in kubertenes/GKE, i have jobs run more than 44 hour, dont have error alert in slack or timeout from Airbyte

I created one guide to show how to restore a deleted connection: https://discuss.airbyte.io/t/how-to-restore-a-deleted-connection/3524

Is there a reliable way to update the source and destination definition yaml files and force a refresh without starting AirByte from scratch or upgrading?

Hi I

Hi Airbyte Team, I'm from KX, a Time series database (kdb+) firm - https://kx.com/ I'm exploring options for a robust Integration tool that can tether popular sources of data-at-rest and data-in-motion to kdb+ Do we need to get into a partnership agreement to make kdb+ as a Data destination to Airbyte? Any pointers?

Hi Everyone, have synced airbyte with airflow locally however while running a dag in airflow, it fails with an exception after making a POST api call to airbyte

[2023-01-03, 07:08:39 UTC] {base.py:73} INFO - Using connection ID 'airbyte_conn' for task execution.
[2023-01-03, 07:08:39 UTC] {http.py:150} INFO - Sending 'POST' to url: <http://localhost:8001/api/v1/connections/sync>
[2023-01-03, 07:08:39 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV

The airbyte connection has S3(cloud) -> Postgres(local) connection and works fine when synced manually. I’m not being able to sync it via airflow. Moreover, the API call to 8001(webapp) server of airbyte requires a basic auth but I didnt find any config to put those creds in airflow. The DAG is as follows -

from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator

with DAG(dag_id='airbyte_job',
         default_args={'owner': 'admin'},
         schedule_interval='@daily',
         start_date=days_ago(1)
         ) as dag:

    airbyte_trigger = AirbyteTriggerSyncOperator(
        task_id='airbyte_airflow',
        airbyte_conn_id='airbyte_conn',
        connection_id='****',
        asynchronous=False,
        timeout=3600,
        wait_seconds=3)

Another possible way to sync it is making an API call from airflow -> airbyte without using airflow’s airbyte provider(https://pypi.org/project/apache-airflow-providers-airbyte/) Can somebody give a curl for running the sync on airbyte?

Hi folks, I am trying to indicate this CRON expression "0 0 14 * * 7" however it is returning this error when saving:

The provided configuration does not fulfill the specification. Errors: invalid cron expression

What could be wrong?

Hi folks, would there be any way to know what is the first synchronization? My idea is that the first time I do a bulk load and then daily only load what has changed. Thanks!

Hey, question about the helm charts: in many places the

fullname

var is being used to which includes a

trunc 63

to prevent creating anything with names that are longer than allowed by k8s, though this is not being used for service names, in this case

{{ .Release.Name }}-airbyte-connector-builder-server-svc

(which is already really long without the release name 😅) I only really see it being used in the env-configmap and would like to know if and how we could possibly work on a solution. More context to follow in thread:

Hi Guys, I am failing to replicate a MYSQL DB table with about 200 Million records. I have hosted Airbyte on AWS EC2 instance (Linux) with the following Specs Type: m5.8xlarge vCPU: 32 RAM: 128GB Is there a way I can increase the replication speed?

[Kubernetes] Hi everyone. I'm having some timeout issues trying to setup a new connection in a newly k8s deployed Airbyte. Airbyte's request to the

check_connection

endpoint times out after 3 seconds. I've logged the issue here: https://github.com/airbytehq/airbyte/issues/20963. Any help is appreciated.

Hi team! I'm trying to test a custom modified connector (ga4) to make a contribution in the near future, BUT! I keep getting the following exception

raise ValueError(f"Unexpected type for data_or_message: {type(data_or_message)}: {data_or_message}")
ValueError: Unexpected type for data_or_message: <class 'collections.ChainMap'>: ChainMap({...},{...},{...})

inside this

collection.ChainMap

is all the data I need to be emitted, with correct values. I just modified a line in the

ga4

connector, where in the stream_slices method I add a

start_date = start_date - datetime.timedelta(days=2)

to make sure the data is golden as per this article