Hi all, I am a beginner trying to follow the tutorial for creating a connector for http API, and I am stuck, I cant install dependenncies for Airbyte CDK, can you please help me fix it :) I get this error :

Obtaining file:///home/cber/airbyte/airbyte-integrations/bases/source-acceptance-test (from -r requirements.txt (line 1))
Obtaining file:///home/cber/airbyte/airbyte-integrations/connectors/source-python-http-example (from -r requirements.txt (line 2))
Collecting PyYAML~=5.4
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
ERROR: Could not find a version that satisfies the requirement airbyte-cdk~=0.1.56 (from source-acceptance-test==0.0.0->-r requirements.txt (line 1)) (from versions: 0.1.0rc3, 0.1.0rc4, 0.1.0rc5, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6rc1, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10, 0.1.11, 0.1.12, 0.1.13, 0.1.14, 0.1.15, 0.1.16, 0.1.17, 0.1.18, 0.1.20, 0.1.21, 0.1.22, 0.1.23, 0.1.24, 0.1.25, 0.1.26, 0.1.27, 0.1.28, 0.1.29, 0.1.30, 0.1.31, 0.1.32, 0.1.33, 0.1.34, 0.1.35, 0.1.36, 0.1.37, 0.1.38, 0.1.39, 0.1.41, 0.1.42, 0.1.43, 0.1.44, 0.1.45, 0.1.46, 0.1.47, 0.1.48, 0.1.49, 0.1.50, 0.1.52, 0.1.53)
ERROR: No matching distribution found for airbyte-cdk~=0.1.56 (from source-acceptance-test==0.0.0->-r requirements.txt (line 1))

Hello Guys! I'm trying to ingest Kafka messages but i always receive "Failed to fetch schema. Please try again". Anyone knows any tutorial or use case that uses Kafka source connector? I tryied on a Kafka running local as well and i received the same response.

Hello all, Needs urgent help; quick response will be appriciated. I've upgraded airbyte version in EKS from

0.35.31-alpha

to

0.39.17-alpha

, by following the guide given here. Essentially, I've pulled the latest code from master branch, provided all the required details in

.env

and

secrets

file and run

kubectl apply -k kube/overlays/stable

. All pods at the start run successfully i.e. there is no restart. SERVER, TEMPORAL and Worker pod's logs (files attached below) are convincing and they don't show any errors. But I'm not able to port-forward

airbyte-webapp

service locally as It is breaking connections (screenshot attached below). When I see the logs of

server-pod

, I see some of the api calls are failing (screenshot attached below). Port-farward command

kubectl port-forward svc/airbyte-webapp-svc 8088:80 -n dev

PS: I"m using RDS postgres instance and S3 bucket. Airbyte is able to access postgres and S3 bucket.

Hello All! I am using open source Airbyte and trying to set up an s3 destination. I was able to setup S3 as a source no problem, but with the destination I get the following error:

Could not connect to the S3 bucket with the provided configuration. Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 957Y2B54792WRDA5; S3 Extended Request ID: tDwOZ1RwjxmSWqZEwJgnOf/znv31wes1vIrfX3QoiJO0Fuyyowm0ji5qyZwjw7nbll927GuSVLA=; Proxy: null)

hello All!. I am having a problem with zendesk support for the metric ticket table because it is giving me "Response Text: Pagination limit reached for offset model" problems, has anyone been able to solve it and can tell me how to fix the problem?

Hello everyone, I’m having some problems with Recurly replication

recurly.base_errors.ApiError: Unexpected 524 Error. Recurly Request Id: None

How could I approach this?

Hi, I’m following the steps to build Airbyte for local development to be able to test a change to a connector I’ve been working on. However, when running

SUB_BUILD=PLATFORM ./gradlew build

I get

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':spotlessJava'.
> java.lang.reflect.InvocationTargetException

This is happening even on master for me. Any help is appreciated

I followed this blog for setup => Set up a modern data stack with Docker

2022-06-12 12:01:25 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-file:0.2.5 exists...

2022-06-12 12:01:25 INFO () LineGobbler(voidCall):85 - airbyte/source-file:0.2.5 was found locally.

2022-06-12 12:01:25 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/521b7c45-1986-42f8-9c25-33da4065a20b/0 --network host --log-driver none airbyte/source-file:0.2.5 check --config source_config.json

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 - Traceback (most recent call last):

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/bin/base-python", line 8, in <module>

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -     sys.exit(main())

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 135, in main

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -     launch(source, sys.argv[1:])

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 120, in launch

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -     AirbyteEntrypoint(source).start(args)

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.7/site-packages/base_python/entrypoint.py", line 90, in start

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -     raw_config = self.source.read_config(parsed_args.config)

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -   File "/usr/local/lib/python3.7/site-packages/base_python/integration.py", line 60, in read_config

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 -     with open(config_path, "r") as file:

2022-06-12 12:01:29 ERROR () LineGobbler(voidCall):85 - FileNotFoundError: [Errno 2] No such file or directory: 'source_config.json'

2022-06-12 12:01:29 INFO () TemporalAttemptExecution(lambda$getWorkerThread$2):149 - Completing future exceptionally...

io.airbyte.workers.WorkerException: Error while getting checking connection.

Getting the above error when uploading file in the source tab. Kindly help.

Hi all, I ran

docker-compose down

command to end airbyte but I am getting following error.

dskim@AD:~/airbyte$ docker-compose down
WARNING: The RUN_DATABASE_MIGRATION_ON_STARTUP variable is not set. Defaulting to a blank string.
WARNING: The NEW_SCHEDULER variable is not set. Defaulting to a blank string.
WARNING: The SECRET_PERSISTENCE variable is not set. Defaulting to a blank string.
WARNING: The WORKER_ENVIRONMENT variable is not set. Defaulting to a blank string.
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

I ran it again with --verbose to obtain debug information.

dskim@HPC01:~/airbyte$ docker-compose --verbose  down
compose.config.config.find: Using configuration files: ./docker-compose.yaml
WARNING: compose.config.environment.__getitem__: The RUN_DATABASE_MIGRATION_ON_STARTUP variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The NEW_SCHEDULER variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The SECRET_PERSISTENCE variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The WORKER_ENVIRONMENT variable is not set. Defaulting to a blank string.
compose.cli.docker_client.get_client: docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
compose.cli.docker_client.get_client: Docker base_url: <http+docker://localhost>
compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '20.10.8', 'Details': {'ApiVersion': '1.41', 'Arch': 'amd64', 'BuildTime': '2021-07-30T19:52:16.000000000+00:00', 'Experimental': 'false', 'GitCommit': '75249d8', 'GoVersion': 'go1.16.6', 'KernelVersion': '5.4.0-94-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.4.9', 'Details': {'GitCommit': 'e25210fe30a0a703442421b0f60afac609f950a3'}}, {'Name': 'runc', 'Version': '1.0.1', 'Details': {'GitCommit': 'v1.0.1-0-g4144b63'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=20.10.8, ApiVersion=1.41, MinAPIVersion=1.12, GitCommit=75249d8, GoVersion=go1.16.6, Os=linux, Arch=amd64, KernelVersion=5.4.0-94-generic, BuildTime=2021-07-30T19:52:16.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('airbyte_default')
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {'1aa21ab12867c91509d016b15a8d2aa664fa7b2b20e89de6d3dc8a7904f229b8': {'EndpointID': '8a482c6e9f762174780ba8b9b918f588668204442429dbc330bc93301f532749',
                                                                                     'IPv4Address': '192.168.112.6/20',
                                                                                     'IPv6Address': '',
                                                                                     'MacAddress': '02:42:c0:a8:70:06',
                                                                                     'Name': 'airbyte-temporal'}},
 'Created': '2022-06-10T22:17:47.71850444+09:00',
 'Driver': 'bridge',
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('airbyte_db')
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume -> {'CreatedAt': '2022-06-11T13:45:43+09:00',
 'Driver': 'local',
 'Labels': {'com.docker.compose.project': 'airbyte',
            'com.docker.compose.version': '1.29.2',
            'com.docker.compose.volume': 'airbyte_db'},
 'Mountpoint': '/var/lib/docker/volumes/airbyte_db/_data',
 'Name': 'airbyte_db',
 'Options': None,
 'Scope': 'local'}
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('airbyte_data')
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume -> {'CreatedAt': '2021-08-27T10:50:51+09:00',
 'Driver': 'local',
 'Labels': {'com.docker.compose.project': 'airbyte',
            'com.docker.compose.version': '1.29.2',
            'com.docker.compose.volume': 'airbyte_data'},
 'Mountpoint': '/var/lib/docker/volumes/airbyte_data/_data',
 'Name': 'airbyte_data',
 'Options': None,
 'Scope': 'local'}
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('airbyte_workspace')
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume -> {'CreatedAt': '2022-06-11T09:17:58+09:00',
 'Driver': 'local',
 'Labels': {'com.docker.compose.project': 'airbyte',
            'com.docker.compose.version': '1.29.2',
            'com.docker.compose.volume': 'airbyte_workspace'},
 'Mountpoint': '/var/lib/docker/volumes/airbyte_workspace/_data',
 'Name': 'airbyte_workspace',
 'Options': None,
 'Scope': 'local'}
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=airbyte']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('1aa21ab12867c91509d016b15a8d2aa664fa7b2b20e89de6d3dc8a7904f229b8')
ERROR: compose.cli.errors.log_timeout_error: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

I can't kill airbyte-temporal container with

docker stop or docker kill

command. how can I solve this? please help.

Hi! 👋 It seems that normalized tables get their names truncated to 43 characters when they have a prefix. Is that an expected behavior? (More details in thread ⬇️ )

Any plans to build authentication support into Airbyte (self hosted)? Seems rather weird that the out-of-the-box setup is wide open

Hello everyone,! I know that airbyte doesn't currently or have plans to support deployment to heroku. But, I was wondering if anyone had managed to deploy using the

heroku.yml

file? Thanks!

Hi Team, does Airbyte deployment on Ec2 support CDC for SQLserver?

The announcement reads they will show how to deploy airbyte on k8s, but already for a long time there’s a bug preventing airbyte from running custom transformations on k8s

Hi guys, Please help us solve this issue: https://github.com/airbytehq/airbyte/issues/13661 This is very important for us.

How reliable is Airbyte in EC2?

Hi guys, I’m trying to connect to a Kafka cluster as a consumer. Normally we are using “security.protocol=SSL” and it connects. In Airbyte I have to specify a JAAS config and I wasn’t able to do so. Airbyte doesn’t throw any intuitive exception, just that it wasn’t able to connect. Does anyone know what configurations I need?

This message was deleted.

Hi again, At which point is the table that an Airbyte destination inserts data into created? Is it the responsibility of the destination? Or is it expected to already exist? Thanks

Hey) I have generated template for Java Destination but getting this error while building. Any idea how to fix?

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-python']
> Failed to apply plugin 'airbyte-python'.

Hi everyone, I'm a new user of Airbyte and I didn't achieve to find an answer to this question : Does anyone know if is it possible to replicate data with Incremental Sync - Deduped History between two Snowflake account. I ask this because when i look at the Snowflake source documentation it is mentioned that Incremental - Append Sync is available and not the Deduped History but in the Snowflake destination documentation the Incremental Deduped history is mentioned. So i would like to know which, the source or the destination, is "restricting" the use of the Incremental Sync Deduped history. Thanks a lot 🙏

airbyte diskspace filling up in 80gb ec2 where im just doing 100 mb full refresh of table data each run happens every 15 minutes i think all the historical data is getting stored in the disk but thats just my assumption

finally got my Facebook app approved for advanced features! 🎉 I'd like to document how I got it approved to help others, how can I contribute?

hey folks, does anyone know a way to run airbyte locally on a different port than 8000? i m trying to follow this guide to get started with local dev. any help would be appreciated! 🙏

hello, im facing a problem when deploying on GCP compute engine, followed the guide but stuck on the last step to connect to airbyte. I am getting an error saying that the port 8000 is used by something else, wondering if there is any help on this.

Hi everyone, I just wanna ask why do I get a Sync Mode of Incremental when the Incremental Dedup History is running for the first time?

👋 Hello, team!

Hi All, I have a question regarding integration with one of MySQL flavours - MariaDB. Is there anyone who knows about possibility of using CDC feature with it? My doubt is that MySQL has binlog and CDC included in its own engine, so you get it right away since the very beginning, with MariaDB there's another case as you can turn on binlog on database itself, but when you want to leverage CDC, you need to setup separate service that is placed on another port than database itself. Airbyte doesn't provide a functionality of selecting port for CDC, so it's hard to understand if this is even possible to be done with MariaDB. Glad to hear your suggestions! Ps. Debezium (which u use) itself says, that it needs just a binlog to read and parse, but still it doesn't help me to understand whether it is enough and so far tests are not successfull :<

Does AirByte must have an AWS Access/Secret key? I installed the chart with minio enabled and keep getting

Internal Server Error: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 16F8CC0611B507BB, Extended Request ID: null)

when trying to add new sources