Hi all, I've setup an Okta->BigQuery Airbyte connector and the data dump was successful. However, I'm only getting the last 7 days of logs whereas Okta has a data retention policy of 3 months. Does someone know if it's possible to get those full 3 months of data?

Hey. Does anyone have recommendations on what kind of instance on AWS should one provision for an external Postgres DB?

Hello everyone, i would like to know if there is any option to connect with .pen files or by ssh, 1 day ago, i tried to make a connection to mongo but it was not viable, what could i do in this case?

Hey guys, how are you? hope everything good, Im trying a MySQL connection, the DB is accesible using DBeaver with the same credentials, but airbyte sends the following error:

Could not connect with provided configuration. Error: Cannot create PoolableConnectionFactory (Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)

I have been looking around and I’m sure the db is accesible, so I’m not sure why it’s not connected, also the credentials are okay. Any one knows what might be happening?

Hi, I've looked around but I can't find an obvious answer - is there a way to set up my source connectors declaratively? E.g. through a definition file, even through an API. As opposed to having to manually set them up in the UI. Thank you!

Hey all, I'm using BigQuery as a Destination, and trying to add GCS Staging. The docs on that page mention the following permissions for the bucket, but I don't see that anywhere in the Permissions tab. I have a service account already (and can specify that as a principal). Am I missing something obvious?

Hello everyone! Is it possible to use the BigQuery source connector to ingest a dataset which is in a project I'm not owner of? Namely I'm trying to ingest data from an openly accessible bitcoin dataset and I'm getting the following error :

com.google.cloud.bigquery.BigQueryException: Access Denied: Project bigquery-public-data: User does
 not have bigquery.jobs.create permission in project bigquery-public-data.

Thanks in advance for any hint!

Hi to all! I am new and would like to know if Airbyte is free? Next question, does this tool allow you to analyze the team?

Hi everyone! Is it safe to join the tables that were created after normalization by

_airbyte_field_name_hashid

?

Hi everyone! Has anyone run into memory issue with migrating 20+ million rows of data and taking forever for the data to come over into Snowflake? Is there a fast way of doing this? Anyone familiar with streaming...would this be a solution?

👋 Hey airbyte people! We’re in the process of testing out airbyte to see if we can use it to power our integrations and I was hoping yall could help answer some questions that came up: • We primarily use ECS, but from what I can tell it looks like support for ECS has been blocked for some time. Could you give us an idea of what level of effort it would take to add support for ECS? (i.e. what needs to be implemented, any other blockers aside from spinning up containers, etc.?) • The License FAQ mentions that yall offer commercial licensing that includes support, maintenance, etc. Do you have any more information on those terms you could send over? ◦ related to that - what’s the timeline looking like for self-hosted enterprise? cc: @Yiyang (Heap.io)

Hi There, Generic Question (feature Request maybe?): How can you have differentiate between DEV/UAT/PROD connections? is there a way to group them in corresponding folders? I am hoping someone would have run into this before me. Cheers,

Hey, does Airbyte support authentication and authorization. if not is there any plan to support it in future?

Hey, does airbyte connect to bitbucket?

Hey all, is there a way to force Airbyte to specify the field type on connections? I'm syncing from MySQL to BigQuery, but its using a

string

for the

created_at

field, even though its a

timestamp

in mysql

Hey, I am trying to deploy Airbyte to EKS. One of the first steps we follow in our deployment process is to create a docker image which is then pulled by the k8s pods and a startup script is run. In the case of Airbyte, is there a recommended base image?

Hi guys! I am starting to integrate airbyte with proprietary ETL flow in python, is there any reference where I could check code examples or docs where I could query googleads already-built source programmatically (without having to code the source, just pass parameters like in the UI)?

Is there any other open source data orchestrator tool simpler than airflow to run airbyte as well as dbt end to end.

I have a question to the Metrics Settings. Is there a way to completely disable the metrics data? I guess it’s related to the environment variable TRACKING_STRATEGY, can you point me to the right documentation? Thanks.

hey.. what's the best way to sum a specific field from a nested array? (multiple levels in). In the example below, i want to show the sum of all "freight_charges" in the data array

hello, i'm newbie, can you give me suggestion for transform data with easy?

Hello! Is the cloud waitlist only for customers that have cloud storage in the US region or also for clouds with storage regions in Europe? And when is cloud estimated to be available in US and EU?

👋 I’ve successfully deployed an Airbyte instance on a Digital Ocean Droplet following the deployment guide. However, I’m wondering how to actually secure that instance (prevent Airbyte UI from being accessible to anybody that has the IP)

Hello! I'm interested in writing articles here. Can anyone please help me understand how to proceed/ the next steps?

Hey Everyone, so we have deployed open source version of airbyte on an ec2 instance. Now we are integrating our hubspot account with postgres DB and I see that our hubspot account has close to 4.5 GB of data which takes about 3 hours to sync in total. I see two modes to sync "Full refresh+Overwrite" and "Full rfersh + append". Which one do you think is good in this use case considering the time it takes for full data to sync...I am considering the option below... 1. Full-refresh+append I think creates duplicates which will need to be handled in the postgres for further analysis

Hello! I have some questions about transformation. If the source and destination are the same DB type (e.g., the source is Postgres, the destination is also Postgres), can it be made to have the same column data type? Alternatively, is there a way to receive or check schema (e.g., column type) of the original source DB in the transformation step using the DBT after connection? Thanks.

Hi, Could someone help me in understanding how source/destination credentials are getting saved inside airbyte [ especially on docker/kube deployments ] - Thanks

Hi, This error occurred when I upgraded from 0.29.12-alpha to version 0.35.30-alpha and

docker-compose up

. What's the problem? Please help. Thanks.

Hello, team! I'm deploying Airbyte on EKS. I've gone through this guide. I've following questions, it would be highly appriciated if some can answer them: 1. Should I go with helm chart? or K8S manifests? 2. What are the resource requirements for each component (webapp, temporal service, scheduler etc.)? 3. Worker component is the main component that do heavylifting job. How to benchmark Worker resource requirements? 4. Is there any guidelines on running Airbyte in HA mode? 5. What should be the size of RDS(postgres) instance at the beginning being used as db? Please share any resources/link which can help me. Do mentioned me @Muhammad Imtiaz in the answer. Thanks in advance,

Hi all, has anyone ever had issues with EC2 instances getting into a state of "Instance reachability check failed" when running airbyte? The only way for me to recover is to restart the instance (I can't even SSH in). I'm trying to sync ~50m records from MySQL to BigQuery. If I select only 1 or 2 tables, then it works fine, but as soon as there are more tables, then it falls over. CPU utlization peaks at ~60%. I've tried with a t3.medium and t3.large, and I've tried changing the GCS Staging from 5mb to 50mb. I can see that the

_airbyte_tmp...

tables have been created in BigQuery, and that's it