Redgate, the company behind Flyway, which is the DB migration software we use, has a statement against modern slavery on their website 🧠

Talend is doing a big webinar event tomorrow: https://www.talend.com/connect/?

the airbyte effect 🙂 https://rivery.io/pricing/

Are there certain rules / requirement when posting in #C01MFR03D5W? I'd like to make sure I'm following any suggested guidelines with any possible future requests.

Anyone here knows about some open-source, web based data transformation tool ? I know Airbyte uses DBT but I’m currently searching if there is any “visual” tool. Thanks!

Hi, can I ask what kind of data is Airbyte collecting? I wasn't able to find it anywhere

Not really an Airbyte-related question but was hoping someone could pitch in. Have setup some tighter security on my GCP Compute instance, setting up Identity Aware Proxy and Firewall rules. However it seems like I have blocked egress traffic from the VM, because now it cant seem to be able to run

sudo apt-get update

It cant connect to any external source. I have tried to setup appropriate egress rules in the firewall settings but so far no luck. Would anyone care to enlighten me?

sorry guyz i think its works with a new bucket, it worked after creating a new bucket

Guys, Is any one hosting airbyte on databricks?

Hello everyone! I'm exploring an idea of creating a free Slack app for open source software communities. The app that create a public archive of messages from public channels and allow reading and search messages without joining Slack community. Similar to archive.org, but for Slack organizations. What do you think about that? What is the chance that the Airbyte community will agree to become a beta tester of such the app in the future?

Anyone noticed Airbyte demo day says Dec 15, but when I try to register it says jan 13. So wondering. https://app.livestorm.co/airbyte/airbyte-demo-day?type=detailed

Not really Airbyte-related but could somebody comment on whether the Facebook Marketing API app supports pulling reporting data from multiple account IDs in one job? And if yes, how does one enable that ability. Currently it seems like I only get data from one account

Team, Can we leave s3 credential blank while attaching an instance profile to ec2 While using s3 as destination

Hi there! What is the most efficient way to go about loading log data into BigQuery from a cloud function? Is it better to do it in two step? store in the most cost efficient way, and then replicate the data in BQ (maybe filtered data) in batches.

Hy @Channel Tools I am doing airflow workshop on next Sunday.In this workshop I’ll give you a full introduction into Airflow and you will learn how to create your first DAG. I will go over all the theory behind Airflow, visit the terminology, and start writing our first DAG. After this workshop you should feel comfortable to write, monitor and debug your own DAGs. Prerequisites: • Knowledge: No prior Airflow knowledge is required, but you need to have a basic understanding of the Python programming language. Registration link: https://forms.gle/JnL9fRsLWEAbP3Ep9

I wanted to share a little piece we wrote as part of this whole fixed price vs hourly struggle that affects remote work. We used to hire developers on Upwork a lot in the past. We always had trouble deciding whether to pick “fixed price” or “hourly”. Fixed price is alluring since you only pay for quality work that you are satisfied with. But, it’s unfair to developers since estimating time precisely is a notoriously difficult thing to do. Moreover, we often found ourselves in the position where we kept having to add little bonuses to keep developers motivated to stick to a project rather than abandon it. So, we switched to hiring people by the hour. But we found developers were annoyed that the time tracking software was capturing their screen. And, some developers tracked exaggerated amounts of time to game the system. Inspired by the Pomodoro technique, we came up with a new technique of tracking time, that works well with our existing tools, namely Slack. I wanted to share the idea and get the community’s feedback on it: https://blog.learningdollars.com/2020/03/31/developer-time-tracking-using-slack-beyond-fixed-price-vs-hourly/

hey guys! im a total noobie. got a question, an I use airbyte to send data from a DWH to salesforce (like a reverse etl) ?

@[DEPRECATED] Marcos Marx @Harshith (Airbyte) @John (Airbyte) Is there anything in the works for similar dbt packages for Airbyte connectors similar to what Fivetran offers? Similar to their ad reporting package…

Guys, I got a bit confused with what I have read here and on some websites. Hope, you will help me to deal with my questions: 1) Is Airbyte free when installed on one's own server? If yes, is it going to change some time? 2) Does it provide some data transformation capabilities (e.g., I get something from DB, filter it, and insert it back to another table of DB)? It was mentioned somewhere about dbt, but dbt has only limited trial period. 3) Does it provide conditional routing of pipeline (e.g., if something was returned by sql-query - go that way, do that....)? 4) Can it send emails with csv attachments? 5) Can I migrate pipelines (with all kind of configurations) or the whole Airbyte instance to another server? Is there some sort of "dump"?

Hi Just asking about experience, With CDC, how much overhead will bring to the source DB?

How does Airbyte generate revenue?

I came across someone trying to join this Slack workspace in another forum but they’re being blocked from doing so since the only allowed domain at https://airbytehq.slack.com/signup#/domain-signup seems to be

@zazmic.com

. Is anyone from Airbyte aware of this or is there a workaround? @John (Airbyte) @Marcos Marx (Airbyte)

Hi there! Do yall have an expected timeframe for how long until Airbyte on kubernetes is out of beta?

Hi. Not sure where to ask this; please direct me elsewhere if inappropriate. Has anyone utilized any of fivetran’s

dbt_<platform>

and

dbt_<platform>_source

packages with Airbyte tables? Obviously, a few of the columns won’t match, namely the platform-specific names -

airbyte_emitted_at

etc, I am trying to avoid re-doing work someone else has already thought through, but don’t see any airybte source-specific DBT models/packages on gh, dbthub, etc., nor are there any forks of fivetrans library of dbt packages Has this been discussed anywhere within the Airbyte community? please point me to old convos if this q has already been answered

Hi there! I testing CDC Postgres send data to bigquery and run very well. When I change table adding a new collun I had to update my schema? There is a way that airbyte could automatic identify new fields? (Sync type Deduped + history)

Team, In docs under scaling airbyte, says to start with an instance 4 or 8 and tune gradually Let say if we configure to auto scale ec2 and it created 2 instances with 4 core each so Is 8gb visible to airbyte

Hello everyone, I was wondering if anyone has attempted to use AWS DMS for syncing Postgres Database to S3, and then using Airbyte to sync from S3 into Snowflake? From what I've read AWS DMS adds in the intended operation as I, U or D to indicate whether the data has been inserted, updated or deleted. Could we potentially use this information to sync data to Snowflake ?

Hi Team, I was going through the code base, I have some question regarding scheduling, currently there are list of specific time options for the data sync scheduling job like [1, 2 3 6 ..] hour, I guess current solution doesn't using temporal cron scheduling option for executing workflow, it has its own scheduling mechanism right? Is there is any specific reason behind it?

Hi, as far as I understand AirByte runs each ingestion process in separate docker container. When it is possible to scale vertically to increase number of docker containers and data ingestion capabilities there can be a moment when vertical scaling will be not enough and horizontal scaling will be required. I wonder how reliable your Kubernetis solution at this moment and if it is possible to already use in prod environment? https://docs.airbyte.com/deploying-airbyte/on-kubernetes

Hi, please redirect me if i’m asking in the wrong place. When I need to add a new object to my salesforce extract, it is my understanding that I have to refresh the schema and re-select all of the original object plus the new one and also set the sync type over again. Is there an easier way or is this how it is to be done? Since doing this resets the destination table, we’ll lose incremental history. Am I correct in assuming we’ll just need to copy those tables separately before the reset if we want to maintain that history? Lastly, what is the incremental logic based on with Salesforce? I assumed it would be Created and or Modified date, but i’m seeing a new copy of each record pulled with every extract that is a complete clone except for the airbyte metadata. Is this a misconfiguration on my part?