Hello, is it possible to discover only specific list of streams by names [ interested ones as a filter to API], currently it is taking too much time for the salesforce account !

In our use case for Airbyte we have multiple sources streaming data to one destination. For example, if I want to ingest Google Sheets data and Salesforce data to a custom api, would I have to pay to stream ALL of that data to the custom api or only the Salesforce data? (Salesforce requires OAuth which appears to be an Airbyte Cloud feature).

Hello, what kind of encryption is being applied on top of source credentials as of now ? Is it possible to pass on the custom hash_key for the same? Any third party integration like vault, cloud secret manager are supported ? Thanks !

Hello everyone, Reading the documentation for MS SQL Server, I am wondering if there is any way to connect via Azure AD credentials? The use case is loading data from cloud Dynamics 365, which is Dataverse under the hood... which is MS SQL Server under the hood.

Hey everyone, can someone tell me where airbyte stores the normalization sql script that it runs for each source. In this case I am trying to connect smart sheets and I am running into some errors which I wanted to debug....

Completed with 1 error and 0 warnings:

normalization - 2022-02-17 22:21:53 INFO () LineGobbler(voidCall):82 -

normalization - 2022-02-17 22:21:53 INFO () LineGobbler(voidCall):82 - Database Error in model small_business___impact_data (models/generated/airbyte_tables/enhanced_capital_701/small_business___impact_data.sql)

normalization - 2022-02-17 22:21:53 INFO () LineGobbler(voidCall):82 -   invalid input syntax for type date: "1.0"

normalization - 2022-02-17 22:21:53 INFO () LineGobbler(voidCall):82 -   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/enhanced_capital_701/small_business___impact_data.sql

Hey everyone, I streaming data from Github to Snowflake using github and snowflake connector. When I ran the process, it takes 40 mins to start

syncing stream

process from the source, it seems idle/waiting something. I tried multiple times and in different environment also showing the same result. why does it take 40 mins to start

syncing stream

process ?

2022-02-05 10:16:53 destination > 2022-02-05 10:16:53 INFO i.a.i.d.s.SnowflakeInternalStagingConsumerFactory(lambda$onStartFunction$1):122 - Preparing stages in destination completed AIRBYTE_SCHEMA_AIRBYTE_RAW_TEAMS 
2022-02-05 10:16:53 destination > 2022-02-05 10:16:53 INFO i.a.i.d.s.SnowflakeInternalStagingConsumerFactory(lambda$onStartFunction$1):125 - Preparing tables in destination completed. 
2022-02-05 10:56:30 source > Syncing stream: repositories 
2022-02-05 10:56:33 source > Read 566 records from repositories stream 
2022-02-05 10:56:33 source > Finished syncing SourceGithub

Hi. I ran docker-compose up, everything started no error logs but still i get Cannot reach server. The server may still be starting up. when I visit localhost:8000

this is error log from web app 2022/02/18 014532 [error] 37#37: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "http://172.19.0.6:8001/api/v1/workspaces/list", host: "localhost:8000", referrer: "http://localhost:8000/"

Hell team, I'm trying to install Airbyte on EKS using helm chart. I want to use my RDS postgres db instead of default one which came through helm charts. Documentation says that I can install it. All I've to do is disable default postgres helm chart and provide external db creds in the values.yaml. But the issue is, secret doesn't created with external db creds. Can you please let me know what else I've to do? https://docs.airbyte.com/deploying-airbyte/on-kubernetes#:~:text=planned%20logging%20improvements.-,U[…]0an%20external%20DB,-After%20Issue%20%233605 https://github.com/airbytehq/airbyte/blob/f927dc501d05dc1030d130c3eb5544e5430af6c2/charts/airbyte/values.yaml#L559

hey team, Trying to setup a postgres source but i don't see the settings for SSH. Pardon the very basic issue, but i can't get this to work I saw in the setup guide that there should be an "SSH key authentication" setting somewhere, but i can't see it on my options.

SSH Tunnel Method defaults to No Tunnel (meaning a direct connection). If you want to use an SSH Tunnel choose SSH Key Authentication or Password Authentication.
Choose Key Authentication if you will be using an RSA Private as your secrets for establishing the SSH Tunnel (see below for more information on generating this key).
Choose Password Authentication if you will be using a password as your secret for establishing the SSH Tunnel.

Thoughts? cc @Emmanuel Tarus

Hi all! 👋 I'm new around and have a basic issue. I'm on the introduction path https://docs.airbyte.com/quickstart/set-up-a-connection. I don't know how to check the data of my first sync. So where I should write the verification code "cat /tmp/airbyte_local/json_data/_airbyte_raw_pokemon.jsonl" ? Thanks.

hey Guys! Some days ago it was announced that Airbyte supports materialized views in Postgres however it does not work for me. Could someone please check it again?https://airbytehq.slack.com/archives/C01A4CAP81L/p1642127264041800 https://github.com/airbytehq/airbyte/issues/9012 Thanks!

Hi all! I am trying to get city and location infos from googleads source, but it seems that geographic_report is not coming with it. In airbyte docs, in the Main Table "Geographic", there are segments like geo_target_city, geo_target_district, geo_target_state. Does someone know where can I find this info? Thanks a lot!

hi everyone ! i'm looking for a way to process jsons files that are located on an amazon ec2 server (centos 7) into a postgres db, can anyone help with the configurations needed ?

what should I put in the repository and version and even name

#  - name: airbyte
#    version: "?.?.?"
#    repository: "https://???-???.???.???.???/"
#    condition: airbyte.enabled

Hi! I have some questions regarding S3 staging with a Snowflake destination and Postgres source. The documentation on part size states:

Affects the size limit of an individual Redshift table. Optional. Increase this if syncing tables larger than 100GB. Files are streamed to S3 in parts. This determines the size of each part, in MBs. As S3 has a limit of 10,000 parts per file, part size affects the table size. This is 10MB by default, resulting in a default table limit of 100GB. Note, a larger part size will result in larger memory requirements. A rule of thumb is to multiply the part size by 10 to get the memory requirement. Modify this with care.

• I see that the default value is '5' in my local build, though the documentation says '10MB'. Is this just a typo, or am I missing something? • I want to try syncing a ~1TB table using this method, which would mean that I should increase the part size to ~100MB, correct? • Which container does the memory requirement rule of thumb apply to? I am working in Kubernetes and want to get my resource limits right.

hi guys, trying to deploy airbyte with helm with existing minio instance, and I get in the ui the following error

Internal Server Error: Error creating S3 log client to Minio

and in the worker logs:

Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Malformed escape pair at index 21: <http://minio-service>:%!d(<nil>)

or if I insert 9000:

Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Malformed escape pair at index 21: <http://minio-service>:%!d(float64=9000)

as you can see it is not parsing well the minio port, even when I insert:

9000

or

'9000'

or

"9000"

or nothing which is this error. here’s my values.yaml file: ``````

Hi. Can anyone help me with local deployment? Server fails to start for some reason. See logs. I tried both on Windows 10 and on Linux Mint. On both the server fails to start. I did git clone and then docker-compose up.

when i connect data with mysql, i've got error, how to solve?, I have filled everything correctly.

I;m trying to connect to the pokemon demo source but it stays stucked in “please wait a little bit more” forever. What am I missing?

Hello , I am trying to deploy airbyte into my cluster following https://docs.airbyte.com/deploying-airbyte/on-kubernetes. I am using my own postgres instance hosted on my cloud. All the service & deployments are up & running except temporal service. Temporal service is continously polling for "waiting for postgresql to start up + nc -z 3306 + '[' 1 -eq 0 ']' + echo 'waiting for postgresql to start up' + sleep 1" Above log is showing incorrect port & invalid host. Note : I am using DATABASE_URL inside overlays/stable-with-resource-limits/.env file and keeping other DATABASE_* properties as blank since my server requires SSL connection. Appreciate any help on this ! Thanks !!!

Hello, I tried to deploy Airbyte to EC2 following this article. Eveything is up and running. now, how can I make an API request from another machine towards this instance in EC2?

Hello, quick question about the cloud service. Knowing that it is not available a atm - can we start off with OSS version and seamlessly switch over to cloud in the future?

Hi all, having trouble with initial setup with the Salesforce connector. I was successfully able to load a smaller table (~5,000 rows, ~100 columns) from Salesforce to our Redshift instance, so can verify that the connection to Salesforce & Redshift were successful. However, when I try to sync one of our larger tables (~25,000 rows, ~600 columns), Airbyte says sync was successful but no rows were transmitted. I believe the error is on Salesforce's side as the SOQL query is too complex, as I received a similar error through another ETL service we are trialing and got it to work after reducing the number of fields (this link says 250 columns will likely not allow the query to run>). Is there any way to either 1.) reduce the number of fields that we are querying from Salesforce, or 2.) pull data from a custom report in Salesforce?

Hello, i am following below link https://docs.airbyte.io/operator-guides/configuring-airbyte-db to configure my existing postgres instance. Something like below DATABASE_URL=jdbc:postgresql://HOST:5432/DB?ssl=true&sslmode=require DATABASE_HOST=HOST DATABASE_PORT=5432 DATABASE_DB=DB Temporal POD getting crashed like below : {"level":"info","ts":"2022-02-23T051034.905Z","msg":"Starting server for services","value":"[history matching frontend worker]","logging-call-at":"server.go:110"} Unable to start server: sql schema version compatibility check failed: pq: SSL connection is required. Please specify SSL options and retry.. If i avoid DATABASE_HOST, DATABASE_PORT, DATABASE_DB , temporal pod is not crashing but service inside pod is not running, it keeps on printing logs like it - trying to connect to posgres with invalid port & host combination. + wait_for_postgres ++ echo ++ awk -F , '{print $1}' + server= + nc -z 3306 Requesting some help with this !

I have 5 million more data, do I have to cut it first when moving it to a new place, will it bring down the server?

Hi does Airbyte support data aggregation at source

Hello, Is there a way to add a suffix to the table name at the destination side ? I need to use wildcard in BigQuery for that I need a suffix appended to the tablename.

Hi guys, I am trying to run airbyte locally on my windows machine, but when I try to open the webapp it says: Cannot reach server. The server may still be starting up. I followed the steps in this article: https://docs.airbyte.com/quickstart/deploy-airbyte.

Hi Guys, Quick question, is there a way in airbyte/ or tool recommend to mask customer data to be in compliance with GDPR and move data to a datawarehouse which is out of Europe?