Hi all, I am trying to sync a table of my MySQL database that is in Google Cloud SQL with BigQuery and I get an error. The connection to the database is done through a SSH tunnel and the error received is

java.lang.OutOfMemoryError: Java heap space

because there are very large records because one of the columns is a JSON which can be very large. The table I am trying to sync contains ~1253802 records (43.8GiB) and there are 3.7MB records. Any help can be good, thank you very much.

good morning guys, I got this error when the sync between postgresl -> snowflake

Too many schemas in schema  "PMS_DEV_DATABASE".PMSLEASING! dbt can only get
        information about schemas with fewer than 10000 objects.

I found that I can modify the adapters.sql ( https://github.com/dbt-labs/dbt-snowflake/blob/21a889ba0d813a4b27c4c1c5bae5688adfd20b9f/dbt/include/snowflake/macros/adapters.sql#L142-L149 _

do you know how ? im using docker

Hi team, I have wrote a new source connector in this pr https://github.com/airbytehq/airbyte/pull/21302, would you like to help me to review it? Thank you very much!

Hi there! Simple question! If a create a Postgres CDC, create the replication slot and the publication, the replication slot only take the tables of the publication or take all the tables? Because the sync runs well but the RS size doesnt decrease. Thanks!

Basic question but I'm not seeing anything in the documentation. On the open-source setup. I want to be able to manage users (my team members) and manage an activity log. I haven't seen this in the documentation yet maybe I'm looking in the wrong place? Is it also possible to create sub-accounts so I can have specific team members assigned to specific projects?

Hi Team, I am trying to automate the airbyte CDC process by using http API. My airbyte is installed in AWS EC2 instance. How can I connect API and create soruce/connection/destination runtime using Python or any other suggested approach ? Please advise.

Hello there!

Hello there! I am using an open source Airbyte which runs on Google Cloud Engine and using Bing Ads as a source connector. Right now I am retrieving all client accounts when getting the data, but is there a way to choose a specific client accounts when doing that? According to this website you can choose but I don’t see how: https://airbyte.gitbook.io/airbyte/integrations/sources/bing-ads

I have a question about dynamically generating streams/catalog. I have an MSSQL database source that the base MSSQL Connector cannot interact with. I have a means of auto generating schemas based on the table DDLs. I use this query within Discover and receive back a valid catalog. I am a little confused as to how Airbyte will identify/render the configured catalog. The documentation has you build a discover method, but there is also a need for a "Schemas" folder as well as a "configured_catalog.json" file. Do I still need to have a "configured_catalog.json" file and/or schema folder since I am auto generating the json at runtime? The CLI requires a catalog file to be passed along with "Read", yet I wasn't planning on having a physical file. I have been using the https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/source-azure-table code as a guide along with the Documentation. Also of note, they do not have a schema's folder at all either and their configured_catalog.json is essentially empty. Is there any reason to avoid imitating this approach?

Hi, Following the documentation and trying to build a custom java destination connector. Getting the following error. OS: Windows JAVA: 17 GRADLE: 7.6 Command I ran using powershell COMMAND: ./gradlew airbyte integrationsconnectorsdestination test java conbuild --stacktrace Stack trace are also attached

extreme noob question re: calling the airbyte API with curl

extreme noob question re: calling the airbyte API with curl Hello everyone! Very new to airbyte and have been playing around with it today. I'm trying to call the API to trigger a sync manually but I can't work out the basics of a curl command to the API. Whatever I try from the CLI, I get the same HTML 401 page including the text "_This deployment of Airbyte is protected by HTTP Basic Authentication_" etc. From https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#overview I can see that with the default credentials, the auth header should be

'Authorization': "Basic YWlyYnl0ZTpwYXNzd29yZA=="

, so I've tried various combinations along the lines of:

curl --request POST localhost:8000/api/v1/connections/sync --header "Authorization: 'Basic YWlyYnl0ZTpwYXNzd29yZA=='" --data "{connectionId: abc123-def456-99999-99999}"

Can anyone point out the obvious flaw here? TIA!

Hi I'm working with Airbyte connectors, though I'm using a high end machine, I'm not getting the performance from Airbyte-sources, I researched a little and found that one possible reason could be because Airbyte sources fetches single rows from sources, so the currently we get very low performance i.e. 1-3k rows per second, however I want to go up to 15-20k rows per second. can someone help with this! 🙏

Hi, a small doubt How to accept a null value as well as a valid value for a particular column in dbt models? Eg - a date field can also contain null as well as date/ a double precision field can contain null as well as a number

Hi all, does anyone else experience an issue with Facebook Marketing since today? I have a few connectors to different pages and starting with 5 AM CET they are all failing with the following error message:

Additional Failure Information: Message: Call was not successful Method: GET Path: <https://graph.facebook.com/v15.0/act_571570143877190/advideos> Params: {'limit': 50, 'filtering': '[{"field":"video.updated_time","operator":"GREATER_THAN","value":1668114864}]', 'summary': 'true'} Status: 400 Response: { "error": { "message": "(#100) Filtering field 'video.updated_time' with operation 'greater_than' is not supported", "type": "OAuthException", "code": 100, "fbtrace_id": "A4FgVSd2DKNiluRut7nm2IO" } }

These are connectors that have been working for months. After the first two failed I updated the Facebook Marketing connector from

0.2.79

to the current

0.2.82

but I still get the same error message. My Airbyte version is

0.40.26

and a log file is attached Thanks for your help

Helo and happy friday! I was wondering if someone have developed the Auction Insights stream for google ads source connector and iswilling to share that stream with me 🙂

Heya Airbyte Team, Thanks for all your work and for providing it as open Source. My organisation as an MVP is assessing the use of self-hosted Airbyte to do ETL from API sources. One of the source connectors Redshift, 1. Currently only supports full refresh and incremental coming very soon. I couldn't find any PR I could watch to get an Idea. ANy pointers would be helpfull

Hi team, would like to help me to merge this pr which has been approved. https://github.com/airbytehq/airbyte/pull/21179

Hello, stupid question: I need to connect to a csv file that is in google drive. I´m trying with the file source connector, but I can´t find the google drive storage provider. In explanation I see: In case of Google Drive, it is necesary to use the Download URL, the format for that is

<https://drive.google.com/uc?export=download&id=[DRIVE_FILE_ID]>

where

[DRIVE_FILE_ID]

is the string found in the Share URL here

<https://drive.google.com/file/d/[DRIVE_FILE_ID]/view?usp=sharing>

Hey Everyone! Is anybody using the LinkedIn Pages integration? We've generated the required client id, secret and refresh token for the Oauth 2.0 method but are still getting a source config error despite having valid credentials:

2023-01-13 12:44:30 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2023-01-13 12:44:30 INFO i.a.c.i.LineGobbler(voidCall):114 - 
2023-01-13 12:44:30 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-linkedin-pages:0.1.0 exists...
2023-01-13 12:44:30 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-linkedin-pages:0.1.0 was found locally.
2023-01-13 12:44:30 INFO i.a.w.p.DockerProcessFactory(create):119 - Creating docker container = source-linkedin-pages-check-1e169675-6112-4891-bbf6-b873fa50b3b5-0-iggim with resources io.airbyte.config.ResourceRequirements@694fc127[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-01-13 12:44:30 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1e169675-6112-4891-bbf6-b873fa50b3b5/0 --log-driver none --name source-linkedin-pages-check-1e169675-6112-4891-bbf6-b873fa50b3b5-0-iggim --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=airbyte/source-linkedin-pages:0.1.0 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.40.18 -e WORKER_JOB_ID=1e169675-6112-4891-bbf6-b873fa50b3b5 airbyte/source-linkedin-pages:0.1.0 check --config source_config.json
2023-01-13 12:44:30 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0
2023-01-13 12:44:33 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):116 - Config validation error: '****' is not of type 'integer'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 123, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 96, in run
    check_config_against_spec_or_exit(connector_config, source_spec)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/utils/schema_helpers.py", line 160, in check_config_against_spec_or_exit
    raise Exception("Config validation error: " + validation_error.message) from None
Exception: Config validation error: '****' is not of type 'integer'
2023-01-13 12:44:33 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):98 - Error checking connection, status: Optional.empty, exit code: 1
2023-01-13 12:44:33 INFO i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling...

Hi! We've got pretty big problem with Airbyte's MS SQL connector. It seems we are not able to replicate a table that we have not allowed it to read some of the columns.. I.e.

GRANT SELECT
    ON dbo.my_table (
                             id,
                             created_at,
                             updated_at,
                             text            
    ) TO airbyte_user;

This same table also has an

email

field, that we do not want to replicate.. and we have not included it in the grant above for obvious reasons. We then receive following error in the logs

"stacktrace" : "java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerException: The SELECT permission was denied on the column 'email' of the object 'my_table', database 'my_database', schema 'dbo'.

How can we trick airbyte into not doing blunt

*

query? Stitch has absolutely no problems with this as it will replicate only the fields the user has told it to replicate.

Hi Everyone. Has anyone got documentation or a link to documentation for deploying airbyte server behind a proxy ? Thank you in advanced Aidan

hi, i’m trying to connect my stripe account with airbyte with a restricted token that has permission for only a subset of data ( disputes )

but i CAN’T save this token on the Airbyte UI because it validate ALL the permissions on the token

Hello 🙂 I upgraded Airbyte from

0.40.22

to

0.40.28

this morning but I have some issues with the temporal service (Example in airbyte-server logs :

Caused by: java.lang.RuntimeException: Could not create Temporal client within max timeout!

) . Did I forget to do something ? (in config or …)

Hi all, I am on my way to try to use the Airbyte open API and I was wondering if there is some more relevant ressources than https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.htm? Indeed some tutorials or examples would be highly appreciated. Thx!

Hi, does anyone know where I can find the possible status responses for a sync job via /jobs/get_light? I'm calling this endpoint

<https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/jobs/get_light>

and so far I've found: • pending • cancelled • succeeded • failed ...any others? Thanks!

Hi, has anybody tried to install octavia-cli added to the airbyte docker-compose.yaml? The installation works fine for me on linux, but the

docker-compose run octavia-cli init

gives an error that the connection with localhost:8000 is refused. Any experiences with connecting octavia internally? A seperate installation with docker on the same server works, but not when included in docker-compose. Thanks!

Error: Could not reach your Airbyte instance, make sure the instance is up and running and network reachable: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f13eca43e50>: Failed to establish a new connection: [Errno 111] Connection refused'))