Hello everyone! To kick-start this channel I would love to know what is the biggest challenge you are facing today when ingesting data into the data warehouse?

Why some connectors available on the website are not in the application?

Hey team I also posted this in the #airbyte-for-beginners channel. I am setting up an

airtable

connector for my marketing team, and I am running into an issue when specifying the tables within the base to pull in. When I specify the table name I get the

HTTPSConnectionPool(host='<http://api.airtable.com|api.airtable.com>', port=443): Max retries exceeded with ...

error. However, when don't specify the table the source in created (and ultimately connector), however, no data pulls in. Many thanks for the assist.

Is anyone here familiar with the BambooHR connector? Running into some issues and not sure who to reach out to, would love to chat with someone who has had experience in setting up the API key from BambooHR and limiting certain fields and how that translates downstream

Hello, I am trying to get some feedback around the source of hubspot and more or less best configuration that you manage to make it work. Especially all around contact (contact table & property_history) the latest never had success to synch it on my BigQuery dataset. Maybe something in the configuration (using incremental for this one) but no success (normalized or denormalized) ? If you have any advice on return on the hubspot ingestion I am interested to hear about 🙂 (hope it’s the right place to post on this topic ).

Hey everyone! We’re using airbyte

0.35.56-alpha

with postgres source connector

0.4.9

with CDC to replicate to bigquery as the destination. We’re observing the same replication problem as - https://github.com/airbytehq/airbyte/issues/8830, where if a row is deleted in source, the destination continues to display the row in the replicated table. The

<table_name>_scd

shows

_is_airbyte_active

column which reflects the deletion, which indicates that there’s a bug in the transformation logic that builds the main table. This is a blocker for us and would be happy to open a PR, if pointed in the right direction. Any advice?

Hey team. Looking for some help with errors on the Postgres Connector. I’ve upgraded to the latest airbyt version and am running the last Postgres Connector Version. Looking at the logs for the sync, it appear to run successfully, but no records are loaded

0 Bytes | no records | no records

. Has anyone experienced this issue?

022-04-06 21:47:45 normalization > 21:47:45  Completed successfully
2022-04-06 21:47:45 normalization > 21:47:45  
2022-04-06 21:47:45 normalization > 21:47:45  Done. PASS=24 WARN=0 ERROR=0 SKIP=0 TOTAL=24
2022-04-06 21:47:45 INFO i.a.w.DefaultNormalizationWorker(run):69 - Normalization executed in 4 minutes 55 seconds.
2022-04-06 21:47:45 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-04-06 21:47:45 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating...
2022-04-06 21:47:45 WARN i.a.s.p.JobNotifier(notifyJob):123 - Failed to successfully notify success:

👋 Hello everyone! Looking for some help with the Stripe Connector on Airbyte Cloud (evaluating now). I first tried with everything on to just test it out, but it failed mysteriously, so I reduced the scope to only grab the customers data. For some reason I’m only seeing “565 committed records” over in Big Query whereas a straight export from the Stripe dashboard shows there are 1248 customers. Is this Stripe source connector being used successfully by anyone else?

When I choose

custom_format

is the schema in my data warehouse supposed to be what i have under

namespace_custom_format

or

destination_stream_prefix

? I'm seeing the data ingesting to schema

mailchimp_

and with a prefix

_airbyte_

and

_airbyte_raw_

For a given source, is there some way to know what the schema of the tables will be , before actually connecting to that source and doing a sync?

Hi everyone! Testing out Airbyte (awesome work!) and running into an issue ingesting data from Redshift. It looks like the connection is getting closed and the sync then hangs in the running state. It appears to do be doing this at around 37 million records, or could also be a timeout setting. Anyone have any ideas on how to handle this? Trying to load billions in so I imagine there’s going to be lots of connection problems during the initial load. Would be great to be able to handle these and pick up where it left off.

2022-04-08 02:16:20 source > 2022-04-08 02:16:20 ERROR i.a.c.u.CompositeIterator(close):93 - exception while closing
2022-04-08 02:16:20 source > java.lang.RuntimeException: java.sql.SQLException: [Amazon](500150) Error setting/closing connection: Not Connected.

Hello! If the company is highly into GCP, is Airbyte a great choice for Data Ingestion? Does Google have it's own Data Ingestion tool?

good morning team Anyone managed to upload the airbyte with EKS - Fargate on Amazon can help me? I have volume related problem

Now that the google sheets connector is going GA and the smartsheet connector is running quite well, has anyone put any thought into an Excel connector for MS graph? Curious if I'm starting up on work someone's already put ground on. https://docs.microsoft.com/en-us/graph/api/resources/excel?view=graph-rest-1.0

Hello, Considering a line from announcements: • The GCS destination connector can now handle more streams. Also, CSV and JSONL formats are now automatically compressed with GZIP. (#11686) Is there the option of uploading still not compressed? I have some external tables that read data from JSONL files from GCS storage and that will break if the files arrive in ZIP format. Thank you!

Hey, guys! Since Google Analytics UA is going to be deprecated in 2023, the connector should be updated to work with GA4 Properties. I checked that there are logged issues: https://github.com/airbytehq/airbyte/issues/6326 But they haven't gotten much traction. We are considering to contribute to the development of a custom GA4 connector, but just want to make sure our work does not overlap with you or other contributors. Are you planning on working on it and how can we contribute?

Hey team, for HTTP sources that offer data export mechanism, i.e., usually requires to initiate a POST (response is some export_req_id) and GET /export_req_id to retrieve data in csv or whatever file format, is Airbyte a right choice for building such connectors?

Hi all, any general advice (max start date, Block size) on connections settings Google Ads -> Google Cloud Storage ? The connection is confirmed but fails during synchronization.

Hello everyone - I'm trying to build a custom connection that's really just a fork of an existing python connection. Are there any extra steps required to work from a fork, as opposed to just creating a new custom connection? Or, are there any docs on this topic? I've looked around a bit but it seems to be mostly start from scratch or use a connector from the catalogue.

i'm using latest 0.36.0-alpha and trying to setup square source. working in a local docker environment, and notice the following in the logs when i try to set up the source:

airbyte-worker      | Caused by: io.airbyte.workers.WorkerException: Could not find image: airbyte/source-square:0.1.4

I’m trying to build a Postgres source and I’m not able to see the list of tables when selecting a particular schema. Do we need to alter the schema in any way to be accessible by Airbyte? The

public

schema is accessible, but the data I want to copy resides in another schema.

Hi guys: this Github Ticket was opened in Sep 2021 - it is holding us back from rolling over Airbyte into production. GA4 is an important source for almost everyone with an eCom webshop: https://github.com/airbytehq/airbyte/issues/6326#issuecomment-1102598125 Will this be worked on soon?

Hi Team, I have an issue with Postgres connector. We are use 0.35.62-Alpha, destination - Postgres 0.3.17 The goal is to load data from one Postgres DB to another. Postgres source connector >= 0.4.5 lead to the issue that no tables fetched from the source. I saw an issue from Feb 15th but no plans from Airbyte team. https://github.com/airbytehq/airbyte/issues/10649 Source connector 0.4.4 fetches the schema but connection fails with error:

Unable to find pod to retrieve exit value. Defaulting to  value 143. This is expected if the job was cancelled.

Logs attached. Could you suggest something?

Hi Team, I am setting up Airbyte from docker but getting the below error. Please help me

airbyte-bootloader | 2022-04-20 07:45:07 ERROR i.a.b.BootloaderApp(assertNonBreakingMigration):189 -                ___ ___________________  ________________ _  __
airbyte-bootloader |               /  |/_ __/_ __/ ____/ | / /_ __/ _/ __ \/ | / /
airbyte-bootloader | ________________________  / /| | / /  / / / __/ / |/ / / / / // / / / |/ / ________________________
airbyte-bootloader | /_____/_____/_____/_____/ / ___ |/ /  / / / /___/ /| / / / _/ // /_/ / /| / /_____/_____/_____/_____/
airbyte-bootloader |  __ _______ __________ /_/ _|_/_/ _/_/_/_____/_/_|_/_/_/ /___/\____/_/_|_/ __ __________ __________
airbyte-bootloader |  / / / / ___// ____/ __ \  / _/ | / / __ \/ / / /_ __/ / __ \/ ____/ __ \/ / / / _/ __ \/ ____/ __ \
airbyte-bootloader | / / / /\__ \/ __/ / /_/ /  / // |/ / /_/ / / / / / /  / /_/ / __/ / / / / / / // // /_/ / __/ / / / /
airbyte-bootloader | / /_/ /___/ / /___/ _, _/ _/ // /| / ____/ /_/ / / /  / _, _/ /___/ /_/ / /_/ // // _, _/ /___/ /_/ /
airbyte-bootloader | \____//____/_____/_/ |_| /___/_/ |_/_/  \____/ /_/  /_/ |_/_____/\___\_\____/___/_/ |_/_____/_____/
airbyte-bootloader | 
airbyte-bootloader | -------------------
airbyte-bootloader |  See details below
airbyte-bootloader | -------------------
airbyte-bootloader | 
airbyte-bootloader | 2022-04-20 07:45:07 ERROR i.a.b.BootloaderApp(assertNonBreakingMigration):197 - Cannot upgrade from version 0.27.1-alpha to version 0.36.1-alpha directly. First you must upgrade to version 0.32.0-alpha. After that upgrade is complete, you may upgrade to version 0.36.1-alpha

Hi Everyone, Do we have a connector for Google's Firebase Analytics?

Hi team, I have an issue with SendGrid connector. Here is an error.

403 Client Error: Forbidden for url: <https://api.sendgrid.com/v3/suppression/blocks?start_time=1641024000&end_time=1650479150&limit=50>

Is there anything I can do to avoid this error?

Hi all, I’m looking for a way to provide OAuth access tokens directly to connections. (Discussion)

Hi Team, Is there a way to automatically (override graphic interface) Edit the streams selection. I have maybe 100 objetcs over a list of 5 000 to filter / edit and it is really painfull. I have a replication that is taking sometime. How can I troubleshoot the bottleneck (I guess my laptop/connection) but I d like to go more knowledgebale on debugging

Hi Team, I have an issue with MySQL connector. We are use 0.36.0-Alpha, source - MySQL 0.5.6 The incremental syncs mode are always failing for large tables, whether it's CDC or standard. https://github.com/airbytehq/airbyte/issues/12103

Another question for the team : I am synchronizing a Sql Server db and some of source tables are massive is there a way to set a permanent filter to off set history. Basically a 50 000 000 row table where only records from 2021 ( 5 000 000 are usefull). I d like to have a static filter like ModifiedAt >= 2021 Also ModifiedAt is my curose_field so I imagine there is a way to override the value used by airbyte to go incremental ?