Good afternoon @Chris (deprecated profile) @Harshith (Airbyte), is it possible to clear attempt history without clearing cursor?

I know for dbt in the jobs posting channel they have a template for formatting job postings as a Slack action. Is this something that could be implemented here for the #getting-started #troubleshooting and possibly the #connector-development?

Hey all, trying to evaluate whether this is a valid configuration for GCP: • Airbyte running on GKE • Application secrets stored in GCP Secret Manager • Use Workload Identity & client libraries on GKE to access • Load into job configuration when running Airbyte Has anyone accomplished something similar before? Were there any hard roadblocks when trying this implementation?

Hello guys! Some feedback regarding the Gitlab connector in thread 😉 I got it to work after a few days but the documentation could help a lot.

Hey folks - I was curious to know more about

Powered by Airbyte

and what does the timeline for it look like? I remember it having a mention on the pricing page too earlier but seems like it has been removed now. Found about it in the Airbyte handbook. We are looking to use

Powered by Fivetran

for now but we are unable to figure a pricing model that works for us to absorb the cost we are incurring on our customers’ behalf. Would love to know more about PBA and see if we can be early partners?

Hi guys, I have an inquiries about the airbyte connectors. I saw in here: https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-appsflyer-singer that airbyte already has connector for appsflyer which it's originated from singer , however I can't found that source connector in here: https://github.com/airbytehq/airbyte/blob/master/airbyte-config/init/src/main/resources/seed/source_definitions.yaml also I didn't found it in Airbyte UI (with image attached), can you help me or explain me about this? thanks!

Hi, I have a question regarding version controlling configurations. I see we can use the configurations export feature, but it stores the credentials in clear text. Do we support reading credentials from environment variables? I was trying to put the config files in git, and make a script to call airbyte API to update when we make changes in git. but I dont want to store clear text credentials in git

Just an idea: Some kind of platform to crowdsource connector building, and match this with developers willing to build it. Lets say 3-4 companies have a need for a connector, but it's not urgent enough for Airbyte to build it, and the companies don't have the developer resources to do it themselves. Then this platform could match companies in need with developers who could offer a price estimate, and then crowdsource the creation of the connector, which would subsequently be open sourced . Think this could offer both the cheapest way for companies to make connectors happen, and expand the value of the entire Airbyte ecosystem

Hi, I have a quick question on the basic normalisation step. Today, I'm ingesting a couple of mysql databases to bigquery using airbyte. What I observe is that mysql integer datatypes ultimately end up as floats in bigquery which is a bit unfortunate, especially for integer ids. Is there a way to better retain the original datatypes in the target databases? without having to write manual transformation steps to re-cast?

Hi everyone. I'm looking for a simple way to verify the counts in all of my source's against my target tables. I can't think of a clean way to do this in airbyte, was hoping anyone had any ideas?

Is there an issue or discussion around refactoring

incremental-dedupe

and the ability to avoid having three variations of the same table e.g.

_AIRBYTE_RAW_MYTABLE

,

MYTABLE_SCD

,

MYTABLE

or at least avoid the

_SCD

table. this issue is a seperate discussion on refactoring the full-table deduping

The updated Facebook Marketing API seems to be creating massive log files for me. It hasn't crashed yet but logs thousands of rows of presumably every row where it doesn't match type null.

Request: tagging endpoints in the docs with the version of Airbyte they were introduced in. Especially working with the various search endpoints, this would be super helpful 🙂

Hi, just a suggestion. I am currently working with a postgres-posrgres suggestion and even though the UI gives a message while reading data from the source, but while working with really large amount of data for a production environment, the message " Executing finalization of tables" (the exact message pasted below) Is not followed by any logs for a really long time (like for two hours and counting for a table which is 59 GB in size) for a user it is a very anxious moment cos he has been waiting patiently for often days and he is not being told about anything that's happening in the program. Please add logs for that steps to atleast let the user know of the progress and status of the program. Thanks. 2021-10-20 103516 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-10-20 103516 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):194 - {} - Executing finalization of tables.

Hi, I think it could help people to mention in Known Issues for Kubernetes the custom dbt transformations limitations : https://docs.airbyte.io/deploying-airbyte/on-kubernetes#known-issues

Hi all, any ETA for Kafka as a source? 🙂

right now it is like this

Your connection from S3 version 0.1.6 to Redshift version 0.3.19 succeeded
This was for sync started on Monday, October 25, 2021 at 1:41:31 PM Coordinated Universal Time, running for 17 minutes 28 seconds.

would be nice to know my connection names, as I have multiple sources using the same connector

Hi! I wanted to ask if there was any more documentation in regards of the SAP Business connector or if anyone has experience using SAP with Airbyte. I’m looking at the documentation here https://docs.airbyte.io/integrations/sources/sap-business-one and was not able to find SAP as a source type when trying to create a new source. Thanks!

Hey! Curious is there's a use case for Airbyte as a CDI. If you use a custom event tracking system, Avo, or Iterative.ly, could you use Airbyte to send platform (web, iOS, Android) events to a data warehouse?

Hello! Is there any way to start one sync after another one finishes?

Hi! I noticed that my airbyte connections' runtime keeps slipping back and my suspicion is that because the scheduler starts the job by looking at the end time of the last run as opposed to the start time of the last run. e.g. if the previous run starts at 11:00am and ends at 11:01am, an hourly sync seems to start the job at 11:01am. Is this right? If so, is there a way to change this behaviour to look at the start time of the previous run time instead? Thanks!

Hi All, I noticed that hubspot deal doesn't support incremental update, is it possible to implement it using hubspot v3 api?

Hi! I'm having an issue when trying to sync mixpanel data from over a year ago using the mixpanel connector for airbyte. When trying to select a start date for the sync on airbyte, I'm using 2017-10-17 as the start date, and I've checked to ensure that our mixpanel has data for this date. When I run the sync, airbyte logs show that the start date is exactly 1 year ago, which is not what I wanted. in airbyte, start_date says : "The default value to use if no bookmark exists for an endpoint. Default is 1 year ago.". I can't find any documentation on this "bookmark". When I use the mixpanel API, I can successfully query events from 2017-10-17. Any ideas? Thanks :)

The s3 connector requires:

Access Key ID
Secret Access Key

Can we remove these requirements? We would like to give a role for airbyte so it can access a specific bucket, instead of using users. Due to internal security constraints. Is there any way for us to do this workaround?

Hi! I'm trying to sync Hubspot as a data source and I noticed that for most of the streams, Airbyte only does full refresh instead of incremental. Is there a safeguard implemented (e.g. by setting some delays between each call to query each page) to prevent the API rate limit from being hit?

Is it possible to use the Airbyte's developer account for Salesloft? I requested a sandbox from Salesloft to test out the Airbyte connector and the partner team recommended me to ask you if it's possible to add me.

Hi! Where do we log requests for functionality to be added to an existing connector? For Shopify, the Inventory_item table isn't synced which contains the cost of products in inventory. This is pretty vital in order for margin calculations

Hi, pretty new here.. from this video, it looks like column filtering was a choice, however it seems to have been removed. I see this is something thats being worked on now, and its def very important, with PII and privacy concerns. Look forward to those changes being implemented.

I'm happy to create an issue for it and possibly work on this if the level of effort appears low (I'm a relatively new Java developer) and the feature makes sense, but it would be great if Airbyte didn't send a message out on the failure alerts webhook for cancelled jobs. If it is required that the cancellations are sent out on a cancelled job in the webhook notifications it would be great to implement an optional

@here

functionality so that we can opt to have an

@here

appended to true pipeline failures and can ignore posts to the Slack channel for cancelled jobs