Hey guys! The first approval has just happened, a few changes and extra-doc stuff then ready to go!

The PR had a PMC approval, so it's ready to be merged! Releases are made once a month, I'll check when the next one will be

Hello - I have a query on the Airflow integration for Airbyte - when the DAG triggers the Airbyte Job and sense for it's completion does it collects all the logs which Airbyte generates underneath and show it on the Airflow UI or it's just sense for state and move forward

I'm having issues with the basic setup of the airbyte_conn_example on cloud composer. I don't appear to be able to connect to the GCE running airbyte. Any help would be appreciated

Following along with these instructions: https://docs.airbyte.io/operator-guides/using-the-airflow-airbyte-operator I have airbyte and airflow on the same docker hosts. They are responding as expted, Airbyte on :8000, Airflow on :8080. I have setup the connection as shown within airflow, and used the sample code for the dag, but I am getting this in my logs, any ideas on what might be wrong?

[2021-07-13 18:27:43,946] {base.py:69} INFO - Using connection to: id: airbyte_conn. Host: localhost, Port: 8001, Schema: , Login: , Password: None, extra: {}
[2021-07-13 18:27:43,947] {http.py:140} INFO - Sending 'POST' to url: <http://localhost:8001/api/v1/connections/sync>
[2021-07-13 18:27:43,950] {http.py:195} WARNING - HTTPConnectionPool(host='localhost', port=8001): Max retries exceeded with url: /api/v1/connections/sync (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8015f2a3d0>: Failed to establish a new connection: [Errno 111] Connection refused')) Tenacity will retry to execute the operation
[2021-07-13 18:27:43,962] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/home/airflow/.local/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/home/airflow/.local/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

We’re trying to run Airbyte using Amazon Managed Workflows for Apache Airflow (MWAA) but haven’t been able to make it work. We see the following issues as perhaps the reason it’s not working: • Airbyte and Airflow are running on different hosts • No access to Airflow host since its a managed service We’ve discussed internally the following alternatives: • Making a call to Airbyte’s API through an SSH task. This would mean an additional step in the process that would requiere development. We would like to avoid intermediate steps. • Expose Airbyte’s API and use a python operator but we couldn’t find a way to requiere authentication for the API. I was wondering 1) if this is the proper channel or should we bring this in #troubleshooting , and 2) if anyone in the community has had experience running Airbyte with MWAA? Thanks!

Does anyone know if there's a company out there that has created automation between airflow and dbt?

Hello All, I am new to Airbyte (coming from Fivetran) and I am using Airbyte cloud for some of my new EL jobs. I am looking to add my Airbyte jobs into my Airflow DAGs, as part of my pipelines, but all docs mention using the OSS version with Airflow. Does Airbyte Cloud support orchestration with Airflow? Are there examples of this your can share? Many thanks for the help.

Hi all, I am using Airbyte installed and managed by Plural. It is running on EKS, I was wondering if anyone knows if it’s possible to integrate with Airflow?

has anyone used airflow as the scheduling mechanism? If so, how did the scaling work? context: I'd like to sync the state of several disparate systems with airbyte before running the airbyte data-sync these operations consist of adding/deleting/altering sources and destinations (this is super awesome and easy because of the API connections [absolutely love it thank you so much!!]) However I'd like to trigger my airbyte data-sync only, and directly, after the states sync up I would then like to use the airbyte-airflow operator to trigger/manage a large number of concurrent data-syncs. Questions: First of all, is this possible/are there any glaring issues with this at plan the outset? Secondly, is there a best practice for scaling the worker operations specifically? Are there any kubernetes (ideally EKS) frameworks or patterns for autoscaling exclusively the workers/jobs? Any and all insights would be greatly appreciated, thank you in advance for your time. Am currently loving airbyte, thank you!!

Everyone from Airbyte Team was removed from this channel because @[DEPRECATED] Marcos Marx archived the channel.

archived the channel