Any advice how to debug it to find out the reason?

Hi everyone, I tried slack notifications on airbyte and everything worked fine, the notifications on the slack channel that I created were successful but I have a little problem with the timezone, the timezone that I received on slack doesn’t match that on airbyte. anyone can help ?

hello team, I'm getting used to connectors from googlesheet -> postgres and working just fine for most cases. I'm generally very happy with the syncs but i wonder why the 'full refresh' is implemented by dropping and recreating the tables instead of truncating. I am trying to maintain views on top of the airbyte tables (not using dbt because the view don't change) but they keep getting dropped. Can someone help me understand if i'm configuring this wrong or point me to some documentation? thanks!

any recommended tool for Airbyte API orchestration?

hello, i have a pretty basic question but am having trouble finding an answer in the docs. for sources that use refresh tokens, will airbyte handle refreshes? if so, where in the airbyte database (im running open source so i have full db access but there are alot of tables to search thru) are the new access/refresh tokens stored? (as i would potentially want to be able to use the tokens outside of airbyte as well.) thanks

Is there any recommended way to secure the Airbyte Open Source API so it can't be pinged externally?

I really want to use airbyte for a project of mine but this seems like a gaping hole in the airbyte system, unless there's a way to secure it

i am trying to deploy airbyte in an ec2 and running docker. it that oftentimes, for example after starting running a sync or something, my ec2 becomes almost entirely unresponsive (airbyte api becomes extremely slow, ssh daemon hangs for extremely long, etc). is there a recommended ec2 instance? im running right now on a t3a.small which is usually sufficient for everything. or could these responsiveness issues be being caused by a different issue?

Hi all, looking for a way to configure SSL certificates for dbt normalization using Postgres destination. To provide some context, I am trying to replicate data from Postgres to Postgres on a Google Cloud SQL instance with SSL enabled. I have successfully setup the source and destinations by putting my ssl keys and certificates in /tmp/airbyte_local. However, normalization seems to be failing because it is trying to connect to my database without SSL. Any advice on how to configure the profiles.yml to support my use case?

Could anyone look at his? I'm stuck.

Hi, I am trying to transfer data from excel to postgresql. I sucsesfuly configure the sources (excel from SFTP ) and the destinations (Postgresql). when I try to set the connection with that object, it is generate "Failed to fetch schema. Please try again." when i looked the log files I found the following error. I do not know how to solve that problem. I atteched the log file also. Can you guide me to solve that problem. Regards Edit: Actualy i try to use local file system as a source. I got the same error.

👋 Hello, team!

sobhitakumarsahu@Sobhitas-MacBook-Air airbyte % docker-compose up WARNING: The RUN_DATABASE_MIGRATION_ON_STARTUP variable is not set. Defaulting to a blank string. WARNING: The NEW_SCHEDULER variable is not set. Defaulting to a blank string. WARNING: The SECRET_PERSISTENCE variable is not set. Defaulting to a blank string. WARNING: The WORKER_ENVIRONMENT variable is not set. Defaulting to a blank string. Traceback (most recent call last): File "docker/api/client.py", line 268, in _raise_for_status File "requests/models.py", line 941, in raise_for_status requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/version During handling of the above exception, another exception occurred: Traceback (most recent call last): File "docker/api/client.py", line 214, in _retrieve_server_version File "docker/api/daemon.py", line 181, in version File "docker/api/client.py", line 274, in _result File "docker/api/client.py", line 270, in _raise_for_status File "docker/errors.py", line 31, in create_api_error_from_http_exception docker.errors.APIError: 500 Server Error for http+docker://localhost/version: Internal Server Error ("b'dial unix /Users/sobhitakumarsahu/Library/Containers/com.docker.docker/Data/docker.raw.sock: connect: no such file or directory'") During handling of the above exception, another exception occurred: Traceback (most recent call last): File "docker-compose", line 3, in <module> File "compose/cli/main.py", line 81, in main File "compose/cli/main.py", line 200, in perform_command File "compose/cli/command.py", line 60, in project_from_options File "compose/cli/command.py", line 152, in get_project File "compose/cli/docker_client.py", line 41, in get_client File "compose/cli/docker_client.py", line 170, in docker_client File "docker/api/client.py", line 197, in init File "docker/api/client.py", line 221, in _retrieve_server_version docker.errors.DockerException: Error while fetching server API version: 500 Server Error for http+docker://localhost/version: Internal Server Error ("b'dial unix /Users/sobhitakumarsahu/Library/Containers/com.docker.docker/Data/docker.raw.sock: connect: no such file or directory'") [3425] Failed to execute script docker-compose

im using logical replication in postgres but SELECT slot_name, confirmed_flush_lsn, pg_current_wal_lsn(), (pg_current_wal_lsn() - confirmed_flush_lsn) AS lsn_distance FROM pg_replication_slots; lsn_distance is keep on increasing why is that ?

Hi Team, I am getting below error source: custom connector (Singer based oracle connector) Target: snowflake OS: Ubuntu Error: 2022-06-02 160409 destination > java.lang.IllegalArgumentException: Message contained record from a stream that was not in the catalog. 2022-06-02 160409 destination > catalog: {"streams":[{"stream":{"name":"mytable","json_schema":{"type":"object","properties":{"DOMAIN":{"type":["string"]},"DESCRIPTION":{"type":["null","string"]}}},"supported_sync_modes":["full_refresh"],"default_cursor_field":[],"source_defined_primary_key":[]},"sync_mode":"full_refresh","cursor_field":[],"destination_sync_mode":"overwrite","primary_key":[]}]} , 2022-06-02 160409 destination > message: {"type":"RECORD","record":{"stream":"myschema-mytable","data":{"DESCRIPTION":"AnalyticsConsultant for solution","DOMAIN":"MyDomain"},"emitted_at":1654185848000}} 2022-06-02 160409 destination > at io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer.throwUnrecognizedStream(BufferedStreamConsumer.java:155) ~[io.airbyte.airbyte-integrations.bases-base-java-0.35.59-alpha.jar:?]

Hi Team, I am trying to connect my facebook page id and facebook page access tocken.. getting following error The connection tests failed. "HTTPError('400 Client Error: Bad Request for url: https://graph.facebook.com/v12.0/105373428866900?access_token=EAAOZCwQoSaZB0BAMZARVzMAtpWw5gAeZBeXMyDD1Mu1ZCkMYkwZAu3mZAEZBPcExRnGaDuGjPGlN3aFDHpeKH08tRQMoJyHMZANKvZC4y9SuvyZC0EggzjnwhmuEYc9cIexQ0KTfSQ0rm2SQR70Np5ZArZAo88IwFMgq3R4SyuvZBHKuCnZCpP9h2YqyRzmNWCuZAZCxuKnZBIRZBkRcqt7uq14iN16qdS1&amp;fields=id%2Cabout%2Cad_campaign%2Caffiliation%2Capp_id%2Cartists_we_like%2Cattire%2Cawards%2Cband_interests%2Cband_members%2Cbio%2Cbirthday%2Cbooking_agent%2Cbuilt%2Ccan_checkin%2Ccan_post%2Ccategory%2Ccategory_list%2Ccheckins%2Ccompany_overview%2Cconnected_page_backed_instagram_account%2Ccontact_address%2Ccountry_page_likes%2Ccover%2Cculinary_team%2Ccurrent_location%2Cdelivery_and_pickup_option_info%2Cdescription%2Cdescription_html%2Cdifferently_open_offerings%2Cdirected_by%2Cdisplay_subtext%2Cdisplayed_message_response_time%2Cemails%2Cengagement%2Cfan_count%2Cfeatured_video%2Cfeatures%2Cfollowers_count%2Cfood_styles%2Cfounded%2Cgeneral_info%2Cgeneral_manager%2Cgenre%2Cglobal_brand_page_name%2Cglobal_brand_root_id%2Chas_added_app%2Chas_transitioned_to_new_page_experience%2Chas_whatsapp_business_number%2Chas_whatsapp_number%2Chometown%2Chours%2Cimpressum%2Cinfluences%2Cis_always_open%2Cis_chain%2Cis_community_page%2Cis_eligible_for_branded_content%2Cis_messenger_bot_get_started_enabled%2Cis_messenger_platform_bot%2Cis_owned%2Cis_permanently_closed%2Cis_published%2Cis_unclaimed%2Cis_webhooks_subscribed%2Cleadgen_tos_acceptance_time%2Cleadgen_tos_accepted%2Cleadgen_tos_accepting_user%2Clink%2Clocation%2Cmembers%2Cmerchant_review_status%2Cmessenger_ads_default_icebreakers%2Cmessenger_ads_default_page_welcome_message%2Cmessenger_ads_default_quick_replies%2Cmessenger_ads_quick_replies_type%2Cmission%2Cmpg%2Cname%2Cname_with_location_descriptor%2Cnetwork%2Cnew_like_count%2Coffer_eligible%2Coverall_star_rating%2Cpage_token%2Cparking%2Cpayment_options%2Cpersonal_info%2Cpersonal_interests%2Cpharma_safety_info%2Cphone%2Cpickup_options%2Cplace_type%2Cplot_outline%2Cpreferred_audience%2Cpress_contact%2Cprice_range%2Cprivacy_info_url%2Cproduced_by%2Cproducts%2Cpromotion_eligible%2Cpromotion_ineligible_reason%2Cpublic_transit%2Crating_count%2Crecord_label%2Crelease_date%2Crestaurant_services%2Crestaurant_specialties%2Cschedule%2Cscreenplay_by%2Cseason%2Csingle_line_address%2Cstarring%2Cstart_info%2Cstore_code%2Cstore_location_descriptor%2Cstore_number%2Cstudio%2Csupports_donate_button_in_live_video%2Ctalking_about_count%2Ctemporary_status%2Cunread_message_count%2Cunread_notif_count%2Cunseen_message_count%2Cusername%2Cverification_status%2Cvoip_info%2Cwebsite%2Cwere_here_count%2Cwhatsapp_number%2Cwritten_by%2Cagencies%2Calbums%2Cblocked%2Ccall_to_actions%2Ccanvas_elements%2Ccommerce_merchant_settings%2Cevents%2Cfeed%2Cglobal_brand_children%2Cgroups%2Cimage_copyrights%2Cindexed_videos%2Cinstagram_accounts%2Clikes%2Clive_videos%2Clocations%2Cnativeoffers%2Cpage_backed_instagram_accounts%2Cphotos%2Cposts%2Cproduct_catalogs%2Cpublished_posts%2Cratings%2Croles%2Crtb_dynamic_posts%2Cscheduled_posts%2Csettings%2Ctabs%2Ctagged%2Cvideo_lists%2Cvideos%2Cvisitor_posts')"

Here is the stacktrace 2022-06-03 061308 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/e1a21792-02fa-46be-af34-b8f63f2f2a14/0/logs.log 2022-06-03 061308 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.7-alpha 2022-06-03 061308 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-facebook-pages:0.1.6 exists... 2022-06-03 061308 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-facebook-pages:0.1.6 was found locally. 2022-06-03 061308 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: e1a21792-02fa-46be-af34-b8f63f2f2a14 2022-06-03 061308 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/e1a21792-02fa-46be-af34-b8f63f2f2a14/0 --log-driver none --name source-facebook-pages-check-e1a21792-02fa-46be-af34-b8f63f2f2a14-0-hkhwr --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-facebook-pages:0.1.6 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.39.7-alpha -e WORKER_JOB_ID=e1a21792-02fa-46be-af34-b8f63f2f2a14 airbyte/source-facebook-pages:0.1.6 check --config source_config.json 2022-06-03 061311 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - Check failed 2022-06-03 061312 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling..

Hello, How do I get advanced access for Facebook Marketing to use the Airbyte Connector? Write the Airbyte app everywhere and make a screencast of me going into Airbyte and trying to access the features I want? What do you do if you don't have your own app?

Hello, I tried getting facebook's advance access but declined.Is there any way to add more delay between apis calls (to not make sure I dont exceed limit) or a way to only extract data that was created a day before ?

Is there a guide for the embedded features?

Hi everyone, Is there a channel for installation related discussion?

Hi am trying follow the steps to create prediction on shopify data. But 'shopify' source connector is not able to establish connection to the app instead it is throwing error below:

ConnectionError(MaxRetryError("HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: /airbyte2.myshopify.com.myshopify.com/admin/api/2021-07/shop.json?limit=250&order=updated_at+asc&updated_at_min=2022-01-01 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7efcf250c190>: Failed to establish a new connection: [Errno -3] Try again'))"))

I am following https://airbyte.com/tutorials/implement-machine-learning-for-your-shopify-store-with-mindsdb this tutorial

Hello everyone! I’m trying out Airbyte cloud for PostgreSQL => AWS Redshift pipeline and have couple of questions 1️⃣ I’m looking at Connector Catalog Destinations > Redshift> docs and am confused as to what sort of privilege does the airbyte user on Redshift side require? It seems that this is sufficient

GRANT CREATE ON DATABASE <my_db> TO <my_user>;

before this, it would tell me it lacks privilege on database, but can someone confirm this is all it needs? 2️⃣ How does the PostgreSQL source connector handle the schema changes? I just tried to do some changes to the connector and upon save, it warned me that all of the data in the schema will be removed.. 🤯 3️⃣ How can I tell airbyte to normalize the JSON field? It currently is trying to push our JSONB into a string type field in Redshift. I’ve already set following in the transformations.

Normalized tabular dataMap the JSON object to the types and format native to the destination.

Looking at Connections and Sync Modes Airbyte basic normalization> it mentions dbt

• only available for destinations that support dbt execution.

but there is no mention as to which destinations support this? I skimmed through the docs, but did not find answers to the above questions. Said that, I’m sure I’m missing something. Thanks in advance! 🙏 P.S. https://docs.airbyte.com/quickstart/ => 404

Hello Team, is there a way to set a custom Name for connections?

Hi Team, is this true ? Python source cdk does not support incremental sync. there is one Issue.

Hi Team, I am trying to migrate data from MySql (Amazon RDS) to my local Postres .. but while trying to configure the destination (Postgres) I am getting following error “*Could not connect with provided configuration. HikariPool-1 - Connection is not available, request timed out after 30003ms.*” Am I missing anything ?

i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - Caused by: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

Hi Team, Anyone running airbyte on EKS ?

Hi Team, I'm testing right now Airbyte for database replication with my

MariaDB <> BigQuery

setup and I have a question regarding sync mode. I have configured binlog for my MariaDB database to thrive from CDC functionality and now wondering what's the best/proper sync mode option for me as I cannot find much enough information in docs. Some of my tables have rows that are updated in the meantime and as far as I know, whether I give

source sync mode

to Incremental or Overwrite, both of them will capture this change, but what about destination mode? Is

append

option will update my row/cell where the change has happened or it will append an entire row and I'll end up with duplicate rows? And what's the reason that my setup doesn't let me to use

Deduped history

option?

Hi Team - I'm a first time poster. I want to replicate data between Microsoft SQL Server to Snowflake. I've successfully done this on one database so far. I applied change-data-capture (CDC) to the new database I want to replicate and have 3 tables with CDC enabled. When I get to the replication screen, I don't see anything except 'dtproperties' as a table option. I should see 3 tables. Is there something I'm missing?