Hi Team I’m trying to setup Airbyte on my local and getting issue while setting-up local Postgres as destination.

Could not connect with provided configuration. HikariPool-1 - Connection is not available, request timed out after 30013ms.

Hello everyone, I just deployed a local version of Airbyte on a VM and have a question: Is it possible to activate Users/Login functionality in the non Cloud version? 😕

Hey everyone, did anyone here use Airbyte open source and BingAds? I cannot get it to work allthough I followed these instructions. I get following error:

The connection tests failed.

OAuthTokenRequestException('invalid_client', "AADSTS700025: Client is public so neither 'client_assertion' nor 'client_secret' should be presented.\r\nTrace ID: 7548d8f2-7c02-456f-a00a-744add096300\r\nCorrelation ID: ce766f83-a3a4-4ece-8aca-67337ae17e62\r\nTimestamp: 2022-06-21 122634Z")

Did anyone have had this error and could solve this or knows what the problem is?

Hi, I used to work with Fivetran and now trying Airbyte. Fivetran has a rich repository of dbt models that are useful to get started quickly with their connectors. I used to import them as packages and extend them with custom data models. Is there any similar repository for Airbyte connectors? Thanks!

Hi Guys, When I try to connect to Oracle as source, I got this error for 1 instance while it works for another

Could not connect with provided configuration. Error: unable to create ONS subscriber with properties provided

Caused by: oracle.ons.SubscriptionRefused: service=sys$users (offset 40)

Anything special I need to config in DB server?

how i can find error in this log? i find this,

2022-06-21 16:11:17 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.refresh_token: is missing but it is required, $.credentials_title: must be a constant value OAuth Credentials
2022-06-21 16:11:16 INFO i.a.w.p.KubePodProcess(close):710 - (pod: airbyte / normalization-redshift-sync-1100-0-ixioh) - Closed all resources for pod
2022-06-21 16:11:16 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally...

But i dont know why, source as Hubspot and Destination as Redshift

Hi, I am currently running a connection from MSSQL to Snowflake and I have the following issue: When selecting a couple of tables (tested with 20 our of 120) everything works well, but when I try with a full sync (so all the streams in the sources) I get the following error:

errors: $.type: does not have a value in the enumeration [RECORD, STATE, LOG, SPEC, CONNECTION_STATUS, CATALOG]
2022-06-21 18:03:40 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: {"type":"TRACE","trace":{"type":"ERROR","emitted_at":1.655834620083E12,"error":{"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'SynchroTiers'.","stack_trace":"[io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:75), io.airbyte.db.jdbc.JdbcDatabase.queryMetadata(JdbcDatabase.java:217), io.airbyte.integrations.source.mssql.MssqlSource.getWrappedColumn(MssqlSource.java:145), io.airbyte.integrations.source.mssql.MssqlSource.queryTableFullRefresh(MssqlSource.java:80), io.airbyte.integrations.source.mssql.MssqlSource.queryTableFullRefresh(MssqlSource.java:52), io.airbyte.integrations.source.relationaldb.AbstractDbSource.getFullRefreshStream(AbstractDbSource.java:299), io.airbyte.integrations.source.relationaldb.AbstractDbSource.createReadIterator(AbstractDbSource.java:245), io.airbyte.integrations.source.relationaldb.AbstractDbSource.getSelectedIterators(AbstractDbSource.java:192), io.airbyte.integrations.source.relationaldb.AbstractDbSource.getFullRefreshIterators(AbstractDbSource.java:149), io.airbyte.integrations.source.relationaldb.AbstractDbSource.read(AbstractDbSource.java:122), io.airbyte.integrations.base.ssh.SshWrappedSource.read(SshWrappedSource.java:54), io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:155), io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:107), io.airbyte.integrations.source.mssql.MssqlSource.main(MssqlSource.java:450)]","failure_type":"system_error"}}}
2022-06-21 18:03:41 INFO i.a.w.p.KubePodProcess(close):708 - (pod: airbyte / source-mssql-sync-1636-0-oyvka) - Closed all resources for pod

so it looks like there is a certain stream in my source which causes the whole sync to fail. I checked the logs for further details but cannot find the respective stream which is causing it. What is the best way to troubleshoot and debug further? Appreciate any hint, thank you!

Hi folks, My first message here 🙂 I'm trying to get Airbyte up and running on GCP and use Identity-aware proxy as a way to control user access to Airbyte. Though, the first thing is I'm unsure how to expose Airbyte to public? I thought this is a default. But it seems it is not. I cannot access it through the external IP (that of the Compute Engine instance; firewall configured as required). I can tunnel it through my localhost (it works this way), but is the default configuration such that Airbyte accepts connections only on the local machine? How to bind it with 0.0.0.0, so that it listens to all? Thanks!

airbyte is workspace space size is increasing every day. 5 days once im running the below script through cron [ec2-user@ip-10-1-29-121 airbyte]$ more cleanup.sh #!/bin/bash cd ~/airbyte docker-compose down docker volume rm airbyte_workspace docker-compose up -d

is there any cleaner approach

Hi Team , I am migrating a data from cloud SQL to bigquery .Is there a way by which i can only migrate the incremental workloads and skip syncing all the historical data from the table.

Is there anyway to register Airbyte custom connector without using UI? I am able to register connector through UI. For a time being I am trying to automate this process.

Hi, I’m new to AirByte. I’m trying to pull Facebook Marketing API data. Once clicked on “connect to source” got error back:

FacebookAPIException('Error: 2635, (#2635) You are calling a deprecated version of the Ads API. Please update to the latest version: v14.0.')

I saw an Issue resolved on github for v13. Tried adding api_version=‘v14.0’ to the api.py but didn’t work. what else could be done to successfully connect to the source?

Hi Team, I’m going through the Python CDK Speedrun: Creating a Source and I’m at the step where I should run

python main.py check --config sample_files/config.json

. I get the following error No such file or directory: ‘/[...]/airbyte/airbyte-integrations/connectors/source-python-http-example/source_python_http_example/spec.json’. I have a spec.yaml (as per the tutorial), why is it asking for a spec.json? Any idea what I’m doing wrong?

Just learning about Airbyte. Is there a best \ common practice it relates to csv files or sftp files.. like how does one ensure the same file is not read multiple times? Typically id ingest a file and archive the file. Do we need to do the same with airbyte or do we need to use a job scheduler to move files around so that airbyte doesnt ingest it again.

Has the scheduler been removed from the deployments in the latest airbyte?

This message was deleted.

How do I change the resources limit/requst for the

read

and

writer

pods airbyte creates in k8s?

is there is a way to update schema like adding new tables to sync in airbyte without reseting all tables or at least i don’t want to select all tables to sync again

Hi, We are trialling AirByte Cloud, however we need to add some custom sources. Per the docs, we can add custom operators. However, I don’t see the menu / option on AirByte Cloud. Are custom sources/destinations only available as part of the self-hosted solution?

Is there a way to make Airbyte spawn more

s3 writers

? Currently it looks like it only creates

1

writer

, and for very large tables (

syncing from pg

), this is quite slow.

Hello, I'm trying to do PostgreSQL CDC to Kafka , Our Kafka Cluster already enable kerberos for authentication, I cannot find documentation to setup kafka configuration with kerberos option in kafka target connector. Does anyone have experience to this ?

Hi Everyone, I have a query. If I want to pass a destination cursor fields in configured_catalog.json where I have a nested json. for example: I want to use lastUpdate as cursor filed how would I do that

"meta": {
           
                "lastUpdate": {
                  "type": "string"
                }
           },

Hi, I’m trying to create to set up a bigQuery with AirbyteAPI, How do i specify the destination to bigQuery? How do i specify my ProjetID/dataset location? Is there an example of the ´connectionConfiguration’ in my request body ?thanks

Hey) What are associations in source connectors? How do they change JSON schema and data received?

Hey guys! Do you provide some sort of sandbox accounts for users to check how the source connectors work?

Hi I'm trying to open port 8001 for airflow to trigger airbyte connection as a whole pipelines, since the airflow and airbyte are in different private vpc. What I can do here to be more secure (authorization) when airflow trigger the airbyte connection ?

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Is Salesforce as a destination supported by any chance?

Hi everyone, I am trying to connect GA as a source, but I don`t know wich column should be the primary key of each table, does anyone knows it?