Hi team, I am encountering some issues in deploying helm chart version 0.40.18 in our eks. The first time postgres pod initialez I can see the Logs: The files belonging to this database system will be owned by user “postgres”. This user must also own the server process. The database cluster will be initialized with locale “en_US.utf8". The default database encoding has accordingly been set to “UTF8”. The default text search configuration will be set to “english”. Data page checksums are disabled. fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... UTC creating configuration files ... ok running bootstrap script ... ok sh: locale: not found 2022-10-18 060717.073 UTC [31] WARNING: no usable system locales were found performing post-bootstrap initialization ... ok initdb: warning: enabling “trust” authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. syncing data to disk ... ok Success. You can now start the database server using: pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start waiting for server to start....2022-10-18 060717.697 UTC [37] LOG: starting PostgreSQL 13.8 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit 2022-10-18 060717.698 UTC [37] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432" 2022-10-18 060717.704 UTC [38] LOG: database system was shut down at 2022-10-18 060717 UTC 2022-10-18 060717.708 UTC [37] LOG: database system is ready to accept connections done server started CREATE DATABASE /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/000_init.sql ALTER SYSTEM CREATE DATABASE You are now connected to database “airbyte” as user “airbyte”. 2022-10-18 060717.933 UTC [52] ERROR: role “docker” does not exist 2022-10-18 060717.933 UTC [52] STATEMENT: GRANT ALL ON DATABASE airbyte TO docker; psql/docker entrypoint initdb.d/000 init.sql12: ERROR: role “docker” does not exist The bootloader pod starts and hangs with message: 2022-10-18 060732,982 main INFO Loading mask data from ’/seed/specs_secrets_mask.yaml 2022-10-18 060734 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable CONFIG_DATABASE_USER: ‘airbyte’ 2022-10-18 060734 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: ‘*****’ 2022-10-18 060734 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable CONFIG_DATABASE_URL: ‘jdbc:postgresql://airbyte-db-svc:5432/db-airbyte’ 2022-10-18 060735 [32mINFO[m c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting... 2022-10-18 060735 [32mINFO[m c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed. 2022-10-18 060735 [32mINFO[m c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting... 2022-10-18 060735 [32mINFO[m c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed. 2022-10-18 060736 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1091 - Using default value for environment variable SECRET_PERSISTENCE: ‘TESTING_CONFIG_DB_TABLE’

Hi everyone, we got an interesting problem including a TimescaleDB on top of a Postgres. Our largest table (>200GB) is partitioned on a specific event timestamp column, but not on it's "created_at" column. Now if we create a connector in Airbyte using "created_at" the replication is just terrible, as TimescaleDB could not properly optimize the query and will query all hypertable buckets (for us > 1000). My idea would be to create a more specific descendant of the Postgres Source, where the cursor field AND the hypertable dimension column can be specified. Does this make sense?

Hi, can someone tell me - if I use API to create connections programatically via python script, do I have to fill all the jsonSchema{} about source table columns and cursor fields details , or is there a way to let the Airbyte to sync Catalog automatically? I am able to create a new connection using API, but the Catalog is empty and I have to refresh it manually via UI now. Thank you!

Hi everyone (again), has anyone ever seen something likes this log, while using a Postgres source?

2022-10-18 04:26:03.376 UTC [25558] [IP] airbyte@parkdepot LOG: SSL error: too many key updates
2022-10-18 04:26:03.376 UTC [25558] [IP] airbyte@parkdepot LOG: could not receive data from client: Connection reset by peer
2022-10-18 04:26:03.376 UTC [25558] [IP] airbyte@parkdepot LOG: unexpected EOF on client connection with an open transaction

Hi everyone, I’m facing a Normalize error when I try to sync data with Airbyte and Clickhouse. For Airbyte I’m using this docker-compose file https://github.com/airbytehq/airbyte/blob/master/docker-compose.yaml Any idea about this error? Thank you!

2022-10-18 09:52:25 [32mINFO[m i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
2022-10-18 09:52:25 [42mnormalization[0m > Configuration:
2022-10-18 09:52:25 [42mnormalization[0m >   profiles.yml file [[32mOK found and valid[0m]
2022-10-18 09:52:25 [42mnormalization[0m >   dbt_project.yml file [[32mOK found and valid[0m]
2022-10-18 09:52:25 [32mINFO[m i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
2022-10-18 09:52:25 [42mnormalization[0m > Required dependencies:
2022-10-18 09:52:25 [42mnormalization[0m >  - git [[32mOK found[0m]
2022-10-18 09:52:25 [32mINFO[m i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
2022-10-18 09:52:25 [42mnormalization[0m > Connection:
2022-10-18 09:52:25 [42mnormalization[0m >   driver: native
2022-10-18 09:52:25 [42mnormalization[0m >   host: host.docker.internal
2022-10-18 09:52:25 [42mnormalization[0m >   port: 8123
2022-10-18 09:52:25 [42mnormalization[0m >   user: cuser
2022-10-18 09:52:25 [42mnormalization[0m >   schema: kronos
2022-10-18 09:52:25 [42mnormalization[0m >   secure: False
2022-10-18 09:52:25 [42mnormalization[0m >   verify: True
2022-10-18 09:52:25 [42mnormalization[0m >   connect_timeout: 10
2022-10-18 09:52:25 [42mnormalization[0m >   send_receive_timeout: 300
2022-10-18 09:52:25 [42mnormalization[0m >   sync_request_timeout: 5
2022-10-18 09:52:25 [42mnormalization[0m >   compress_block_size: 1048576
2022-10-18 09:52:25 [42mnormalization[0m >   compression: 
2022-10-18 09:52:25 [42mnormalization[0m >   custom_settings: None
2022-10-18 09:52:25 [42mnormalization[0m >   Connection test: [[31mERROR[0m]
2022-10-18 09:52:25 [32mINFO[m i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
2022-10-18 09:52:25 [42mnormalization[0m > [31m1 check failed:[0m
2022-10-18 09:52:25 [42mnormalization[0m > dbt was unable to connect to the specified database.
2022-10-18 09:52:25 [42mnormalization[0m > The database returned the following error:
2022-10-18 09:52:25 [32mINFO[m i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
2022-10-18 09:52:25 [42mnormalization[0m >   >Database Error
2022-10-18 09:52:25 [42mnormalization[0m >   Code: 102. Unexpected packet from server host.docker.internal:8123 (expected Hello or Exception, got Unknown packet)
2022-10-18 09:52:25 [32mINFO[m i.a.w.n.NormalizationAirbyteStreamFactory(filterOutAndHandleNonAirbyteMessageLines):104 - 
2022-10-18 09:52:25 [42mnormalization[0m > Check your database credentials and try again. For more information, visit:
2022-10-18 09:52:25 [42mnormalization[0m > <https://docs.getdbt.com/docs/configure-your-profile>

Hi All, Can anybody please tell me does Airbyte supports Unstructured and Semi-Structured data replication also?

Hi, Airbyte OSS deployed on EC2. Source Connector: Hubspot Target Connector: Redshift + S3 staging Error: I am facing the following error when running my Hubspot to Redshift connection.

Caused by: java.lang.RuntimeException: java.sql.SQLException: [Amazon](500310) Invalid operation: Problem reading manifest file - S3CurlException: Connection timed out after 50001 milliseconds, CurlError 28, multiCurlError 0, CanRetry 1, UserError 0 
Details: -----------------------------------------------
  error:  Problem reading manifest file - S3CurlException: Connection timed out after 50001 milliseconds, CurlError 28, multiCurlError 0, CanRetry 1, UserError 0
  code:      9001
  context:   <s3://insurely-airbyte-prod/data_sync/prod/hubspot_campaigns/2022_10_18_07_682ec106-c3f7-498b-9f10-ac0fbd3f233d/ff8681dd-d32a-4184-a263-b66f22843c66.manifest>
  query:     2051168
  location:  s3_utility.cpp:334
  process:   padbmaster [pid=18629]
  -----------------------------------------------;
	at io.airbyte.commons.lang.Exceptions.castCheckedToRuntime(Exceptions.java:58)
	at io.airbyte.commons.lang.Exceptions.toRuntime(Exceptions.java:41)
	at io.airbyte.integrations.destination.redshift.operations.RedshiftS3StagingSqlOperations.executeCopy(RedshiftS3StagingSqlOperations.java:137)
	at io.airbyte.integrations.destination.redshift.operations.RedshiftS3StagingSqlOperations.lambda$copyIntoTmpTableFromStage$1(RedshiftS3StagingSqlOperations.java:107)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at io.airbyte.integrations.destination.redshift.operations.RedshiftS3StagingSqlOperations.lambda$copyIntoTmpTableFromStage$2(RedshiftS3StagingSqlOperations.java:107)
	at io.airbyte.commons.lang.Exceptions.castCheckedToRuntime(Exceptions.java:54)
	at io.airbyte.commons.lang.Exceptions.toRuntime(Exceptions.java:41)
	at io.airbyte.integrations.destination.redshift.operations.RedshiftS3StagingSqlOperations.copyIntoTmpTableFromStage(RedshiftS3StagingSqlOperations.java:105)
	at io.airbyte.integrations.destination.staging.StagingConsumerFactory.lambda$onCloseFunction$3(StagingConsumerFactory.java:195)
	... 6 more

Hi we are seeing this issue when trying to pull a json file from zoho

Failed to load <https://zohourlhere>: ConfigurationError('Reader json is not supported\nTraceback (most recent call last):\n File "/airbyte/integration_code/source_file/client.py", line 314, in load_dataframes\n reader = readers[self._reader_format]\nKeyError: \'json\'\n') Traceback (most recent call last): File "/airbyte/integration_code/source_file/client.py", line 314, in load_dataframes reader = readers[self._reader_format] KeyError: 'json' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/airbyte/integration_code/source_file/source.py", line 95, in check next(client.load_dataframes(f)) File "/airbyte/integration_code/source_file/client.py", line 318, in load_dataframes raise ConfigurationError(error_msg) from err source_file.client.ConfigurationError: Reader json is not supported Traceback (most recent call last): File "/airbyte/integration_code/source_file/client.py", line 314, in load_dataframes reader = readers[self._reader_format] KeyError: 'json'

Can someone help please?

Hey all, I am trying to setup airbyte on AWS EKS using manifests files I read the files and I see that there is a deployment of airbyte-cron Can anyone please help me to understand why we need airbyte-cron ?

Having issues connecting airbyte hosted on EC2 to Redshift Message: HikariPool-1 - Connection is not available, request timed out after 60005ms. 2022-10-18 120204 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):99 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'} 2022-10-18 120205 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-18 120205 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-18 120205 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-18 120205 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - The "standard" upload mode is not performant, and is not recommended for production. Please use the Amazon S3 upload mode if you are syncing a large amount of data. 2022-10-18 120205 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):99 - Using destination type: STANDARD

Hey! We are running on Kubernetes over EKS, deployed using he helm chart on version 0.39.41-alpha Every once in a while all our jobs start failing. We see very high memory usage (~4Gi RAM for 4 active jobs) in the worker pod, and after restarting the worker and server the jobs succeed. Unfortunately we lost the logs from the worker pod after we restarted it… Here are the logs from one of the failed sync jobs:

2022-10-18 11:34:34 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):100 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:84) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:63) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:37) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:102) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	... 5 more

any ways to speed up normalization? second with no new data, took almost an hour on just normalization.... If i need this to run every hour, it doesn't seem possible... postgres to snowflake, all running the latest versions, all of these are incremental dedup(hence the large amount of normalizaiton) scaled the snowflake warehouse, which would help a little but anything i can do to speed up normalization other than scale snowflake?

Does anyone have a solution to the storage disk being maxed out after a couple of days?

I want to ingest data from my GitHub repo with airbyte. All my connections are successful between the source (GitHub) and destination (postgres). However, I can't find the data in any of the schema loaded into postgres. Any help with this?

Hello everyone! How can a use the airbyte API to "refresh source schema" of a connection like in the interface?

We using open source Airbyte on Ubuntu; is there a way we can use API or create a python scripts to create sources/destination connectors. any examples will be greatly appericated.

Need Help. Deploying Airbyte on Google Cloud (Compute Engine) using the official docs https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine . I have executed all the commands successfully, with the exception of the final one "# In your workstation terminal gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000localhost8000 -N -f". This throws out the ff error message "ERROR: gcloud.beta.compute.ssh) argument [USER@]INSTANCE: Must be specified ..." .

2022-10-18 14:30:45 normalization > Database Error in model RESTAURANT_SCD (models/generated/airbyte_incremental/scd/READ_MIRROR_V3/RESTAURANT_SCD.sql)
2022-10-18 14:30:45 normalization >   100035 (22007): Timestamp '+192153-11-18T12:06:13.000000' is not recognized

this is in the snowflake destination, which i understand won't support the timestamp... but im not sure where this timstamp is coming from, on the db source, i search or it but can't find it:

SELECT *
FROM restaurant
WHERE updated_at = '192153-11-18 12:06:13.000000';

Is there an easy way to each for the airbyte row id in the raw table in snowflake to see the entire row with the issue?

hey guys, so I was following this tutorial to set up airbyte on GCP https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine/ at the final step,

gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000:localhost:8000 -N -f

I encounter the problem

bind [127.0.0.1]:8000: Address already in use

I have no idea what port is using it. I have tried to change to port 8080; 80; 4444 but none work. What should I do here? thank you

Hi. The Airbyte Snowflake connector has 4 different dataload options (Snowflake’s Internal Stage, AWS S3, GC Bucket, Azure Blob). I was wondering what the benefit would be in not using Snowflake’s Internal Stage but to stage via one of the other storage mechanisms. Would I see any performance/cost benefit if I’m loading from an AWS Aurora MySQL DB to Snowflake and Airbyte is hosted outside of AWS by using the S3 storage data loading method?

Usually I am using Azure Data factory to create my ETL. What's differential between Airbyte and Azure data factory? I ask because I want take to my data lead the Airbyte.

Hi all, we're using helm charts to deploy airbyte. We have an issue with

airbyte webapp

where the deployment ignores serviceAccount. We have

global.serviceAccountName

as well as

serviceAccount:
  create: false
  name:..

and everything's ok for every other pod except webapp. If I view resource description I still see

serviceAccountName: default

while for others I can see the values we've provided. We're using the service account to load secrets from Google Secret Manager (by attaching a volume) and would prefer to not have them hardcoded anywhere as it makes secret lifecycle much simpler for us. Is there anything we can do?

Is e-commerce support available for google analytics by now https://github.com/airbytehq/airbyte/issues/10049 seems to be stale

Hi, Since IBM Db2 is not available as destination we would like to move data into our data warehouse (Db2 Warehouse on Cloud) with a more general destination. Isn’t there anything like general JDBC destination available?

Hi team, We are using Airbyte for our data pipeline to move data from postgress to Clickhouse. We are using Cron for data sync. This is the syntax , using for step time (/5 * * * * ?) . Even, we give 5 as step value, airbyte is taking 1 as step value. It is not taking 5. It is happening, when you give whatever the minute values as step. Requesting help to solve this. Can you guys please give suggestions to solve this?

Hey, I am building a new Airbyte connector using the low-code connector development. I generally follow the tutorial. Running:

python main.py check --config secrets/config.json

returns

{"type": "LOG", "log": {"level": "INFO", "message": "Check succeeded"}}
{"type": "CONNECTION_STATUS", "connectionStatus": {"status": "SUCCEEDED"}}

so the connection works. However at step 4, when running

python main.py read --debug --config secrets/config.json --catalog integration_tests/configured_catalog.json

I get the following error message

{"type": "DEBUG", "message": "Debug logs enabled", "data": {}}
{"type": "LOG", "log": {"level": "FATAL", "message": "Expecting value: line 12 column 3 (char 240)\nTraceback (most recent call last):\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/main.py\", line 13, in <module>\n  launch(source, sys.argv[1:])\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 123, in launch\n  for message in source_entrypoint.run(parsed_args):\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 111, in run\n  config_catalog = self.source.read_catalog(parsed_args.catalog)\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\", line 90, in read_catalog\n  return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/connector.py\", line 53, in read_config\n  return json.loads(contents)\n File \"/Users/sebastianbrickel/opt/miniconda3/lib/python3.9/json/_init_.py\", line 346, in loads\n  return _default_decoder.decode(s)\n File \"/Users/sebastianbrickel/opt/miniconda3/lib/python3.9/json/decoder.py\", line 337, in decode\n  obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File \"/Users/sebastianbrickel/opt/miniconda3/lib/python3.9/json/decoder.py\", line 355, in raw_decode\n  raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 12 column 3 (char 240)"}}
{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1666179423642.493, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "Expecting value: line 12 column 3 (char 240)", "stack_trace": "Traceback (most recent call last):\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/main.py\", line 13, in <module>\n  launch(source, sys.argv[1:])\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 123, in launch\n  for message in source_entrypoint.run(parsed_args):\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 111, in run\n  config_catalog = self.source.read_catalog(parsed_args.catalog)\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\", line 90, in read_catalog\n  return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\n File \"/Users/sebastianbrickel/Documents/airbyte/airbyte-integrations/connectors/source-waiteraid/.venv/lib/python3.9/site-packages/airbyte_cdk/connector.py\", line 53, in read_config\n  return json.loads(contents)\n File \"/Users/sebastianbrickel/opt/miniconda3/lib/python3.9/json/_init_.py\", line 346, in loads\n  return _default_decoder.decode(s)\n File \"/Users/sebastianbrickel/opt/miniconda3/lib/python3.9/json/decoder.py\", line 337, in decode\n  obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File \"/Users/sebastianbrickel/opt/miniconda3/lib/python3.9/json/decoder.py\", line 355, in raw_decode\n  raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 12 column 3 (char 240)\n", "failure_type": "system_error"}}}

I am completely stuck right now. Any advice/hint is welcome

Hello, Where may I find a guide about creating a connector via JS?

Hi team im curious about the usage of connectors and connections via the Airbyte REST API idea : we would like to create a connection as kind of a "template" beforhand (either manually via the gui or via the airbyte api). say source is a rest api from a service and destination are tables in postgres. then we would like to trigger a sync with this connection, but to be able to "override" the config for this this api call (just to be able to call the connection with other arguments) currently, i don't see in the api docs, that this is possible. we would start the sync of the connection via dagster => and there also dagster has only parameters to set the host and post of the airbyte instance and the id of the connection, but no parameters specific for the source. is this somehow possible via api ? because when we test a connector written in python, we then also pass in the config via the command line e.g. "python myconnection.py check --config someconfig.json" => would it be possible to adapt this to a connection via rest api call ?

Hi Airbyte Team! I just launched the local instance of your application in order to do some testing if it will be good fit for our project. As a test data I use SFTP connection with 5 files from a different dates. During my testing I found 2 things that I have no idea how to set: 1. All of those 5 files are saved in the Snowflake database with a different names - so 5 files = 5 tables. How can I force the system to write info a single database table? All the files have the same structure 2. After those 5 files were imported I added 1 more file and triggered the refresh. However that additional file was not imported. Is there any option to always include all files from the selected folder? I added the regex rule while creating the source - so that 6th file should be also added. Thank you very much for your help!!

Hello all - Has anyone found a stable resolution to the "Cannot connect to Server" issue on local deploy? I have the latest build and cannot get the system to start up. Any advice is greatly appreciated. Running: MacBook M1