I am connecting mongodb as a source and airbyte is throwing this err: HELP ME: "Could not connect with provided configuration. Error: java.lang.IllegalArgumentException: The connection string contains an invalid host 'mongodb+srv:'. The port '' is not a valid, it must be an integer between 0 and 65535"

Hello everyone, we are developing a customer connector on Criteo, the endpoint we are trying to access is configured as a POST so we get this error :

{
  "type": "LOG",
  "log": {
    "level": "ERROR",
    "message": "{\"warnings\":[],\"errors\":[{\"traceIdentifier\":null,\"type\":\"unknown\",\"code\":\"not-found\",\"instance\":null,\"title\":\"The path has not been found\",\"detail\":\"The path GET /2022-10/statistics/report does not exist.\",\"source\":null}]}"
  }
}

https://developers.criteo.com/marketing-solutions/docs/campaign-statistics#creating-a-report-request When we tried a different endpoint that is configured as a GET we don't have such error and everything works well, would you have any idea how to modify so that it sends a POST request to the endpoint ? Thank you !

Hi, I am having an issue where duplicates are occurring during a Full Refresh | Overwrite sync. • Deployment method: OSS version 0.40.18 on EC2 • Connection details: ◦ MySQL (connector version 10.0.13) -> Redshift (connector version 0.3.51) ◦ All tables set to Full Refresh | Overwrite ◦ Runs 1 time a day on a CRON Schedule • Error info: ◦ Only in a handful of tables, am I seeing duplicate records in the destination table. ◦ Here is an example where the destination table has a row count of 162 786 rows ◦ When I look at the logs for that specific table, I get the following info about the stream. And this is the same patter for the other tables with the duplicates. What could be causing the duplicates? Both rows have the same Airbyte emitted at datetime stamp:

{
    "streamName" : "anonymous_table",
    "stats" : {
      "recordsEmitted" : 81394,
      "bytesEmitted" : 16462411,
      "recordsCommitted" : 81394
    }

Hi. I am trying to set a Shopify connector but I am getting this message.

Hi folks! We have a multi-tenant MySQL instance where each database (aka schema) have the same structure and contains data for a certain customer. In Airbyte you can only create a connection to a single database as a source. We have thousands customers though. So is there a way to create sources for all the customers dynamically/programmatically? Appreciate any advice.

Hi, I upgraded the MySQL source connector yesterday to version 1.0.13 from a version prior to 0.6.9. I've fixed the source configuration in the airbyte db according to the mysql connector docs (changing the json structure of the

replication_method

config entry) and the sources don't throw any errors when I save them. However, I started seeing connection sync failures with the following error: (full log attached)

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Index 0 out of bounds for length 0
	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:67)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
	at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$0(IntegrationRunner.java:220)
	at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:270)
	at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:219)
	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:141)
	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:100)
	at io.airbyte.integrations.source.mysql.MySqlSource.main(MySqlSource.java:309)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Index 0 out of bounds for length 0
	at io.airbyte.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:75)
	at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
	at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:49)
	at io.airbyte.commons.util.LazyAutoCloseableIterator.close(LazyAutoCloseableIterator.java:52)
	at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:49)
	at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:49)
	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:65)
	... 15 more
Caused by: java.sql.SQLException: Index 0 out of bounds for length 0
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2042)
	at com.zaxxer.hikari.pool.ProxyConnection.setAutoCommit(ProxyConnection.java:401)
	at com.zaxxer.hikari.pool.HikariProxyConnection.setAutoCommit(HikariProxyConnection.java)
	at io.airbyte.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:72)
	... 24 more
Caused by: com.mysql.cj.exceptions.CJException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:124)
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:699)
	at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2033)
	... 27 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
	at com.mysql.cj.protocol.a.NativePacketPayload.readInteger(NativePacketPayload.java:398)
	at com.mysql.cj.protocol.a.NativePacketPayload.readString(NativePacketPayload.java:605)
	at com.mysql.cj.protocol.a.NativeServerSessionStateController$NativeServerSessionStateChanges.init(NativeServerSessionStateController.java:112)
	at com.mysql.cj.protocol.a.result.OkPacket.parse(OkPacket.java:66)
	at com.mysql.cj.protocol.a.NativeProtocol.readServerStatusForResultSets(NativeProtocol.java:1755)
	at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:116)
	at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:48)
	at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1664)
	at com.mysql.cj.protocol.a.NativeProtocol.readAllResults(NativeProtocol.java:1718)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1064)
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:997)
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:663)
	... 28 more

Any idea what could cause this?

Hi All, I am running a airbyte oss in my local, I want to access airbyte db. How can I get achieve that? Where I could see db cofiguration details so that I can access it.

Hello, maybe someone could help me 😞 Description in discuss — https://discuss.airbyte.io/t/failed-bigquery-clickhouse-sync/3217 I can give an additional information if someone tell me what kind of

Hi everyone! We have set up a connection between

Salesforce

and

BigQuery

. We have run it multiples times, with same start date and

Full refresh | Overwrite

as sync mode and we have noticed that the number of

emitted records

has been decreasing. Is this an expected behaviour? If so, could you please explain why this happens? Thanks!

Hi there 👋 I’m having issues setting up a source in S3 where the files have Avro format and gzip compression. The Airbyte version I use is: 0.39.35-alpha The s3 source version is: 0.1.25 And this is the error I see in the logs:

2022-11-22 12:13:20 [1;31mERROR[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - 'avro.schema'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 116, in run
    catalog = self.source.discover(self.logger, config)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 72, in discover
    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 72, in <listcomp>
    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 113, in as_airbyte_stream
    stream = AirbyteStream(name=self.name, json_schema=dict(self.get_json_schema()), supported_sync_modes=[SyncMode.full_refresh])
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 184, in get_json_schema
    for column, typ in self._get_schema_map().items()
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 169, in _get_schema_map
    return_schema = self._get_master_schema()
  File "/airbyte/integration_code/source_s3/source_files_abstract/stream.py", line 234, in _get_master_schema
    this_schema = file_reader.get_inferred_schema(f, file_info)
  File "/airbyte/integration_code/source_s3/source_files_abstract/formats/avro_parser.py", line 79, in get_inferred_schema
    avro_schema = self._get_avro_schema(file)
  File "/airbyte/integration_code/source_s3/source_files_abstract/formats/avro_parser.py", line 65, in _get_avro_schema
    avro_reader = fastavro.reader(file)
  File "fastavro/_read.pyx", line 1134, in fastavro._read.reader.__init__
  File "fastavro/_read.pyx", line 1095, in fastavro._read.file_reader.__init__
KeyError: 'avro.schema'

I get the same error when I try to load the gzipped file with fastavro, so my guess is that Airbyte is not detecting the compression format. Is Avro gzipped supported for S3 source? I could not see how to specify the compression format either. Any help is appreciated, thank you!

Hello everyone, I am trying to build a connectors. All the tests and commands run without issue until this point. To test is locally I ran this command and docker build . -t airbyte/source-impact:dev and tried to add on the UI side but getting an error :

Internal Server Error: Get Spec job failed.

Am I missing something here? Appreciate your help.

Hi all, I have a problem configure Airbyte to write logs to GCS using existing secrets stored in GCP secret manager. Anyone know how to configure this properly? I have deployed Airbyte through Helm and additionally specify SecretProviderClass like follow:

apiVersion: <http://secrets-store.csi.x-k8s.io/v1|secrets-store.csi.x-k8s.io/v1>
kind: SecretProviderClass
metadata:
  name: airbyte-logs-secrets
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/<id>/secrets/data_airbyte-logs-secrets/versions/latest"
        path: "secrets.json"

And in

values.yaml

I specify the logs section as follow:

gcs:
  bucket: "<bucket-name>/logs"
  # If you are mounting an existing secret to extraVolumes on scheduler, server and worker
  # deployments, then set credentials to the path of the mounted JSON file    
  credentials: "secrets.json"
  # If credentialsJson is set then credentials auto resolves (to /secrets/gcs-log-creds/gcp.json)
  credentialsJson: ""

And also mount the volume to worker and server as follow:

extraVolumes:
  - name: airbyte-logs-secrets
    csi:
      driver: <http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io>
      readOnly: true
      volumeAttributes:
        secretProviderClass: airbyte-logs-secrets

However, I’m getting this error

Hello everyone, we just updated our airbyte version to 0.40.22 and we are trying to get data from shopify. However, we see warnings from shopify saying that our airbyte is making calls to deprecated shopify endpoints, namely order and customer endpoints. Did anyone experience this issue before? How did you fix it?

Hello everyone, I am having an issue with data transformation on clickhouse with airbyte, can you please help me or you can connect me with someone who can help, Issue:- When airbyte is migrating my data to clickhouse then my materialized view is not executing, ideally it should trigger just after data gets inserted to to the table.

Hello team 👋 I'm trying to run Stripe ingestions to BigQuery in

Incremental | Deduped history

mode, but the state is never persisted to the DB, even once the connector is in success 😕 The impact is that data syncs always ingest the whole history as if I was in

Full Refresh

mode. Is someone facing the same issue? How do you think I could fix this? Thanks in advance for your help! 😃🙏 Airbyte version:

0.40.22

Stripe source connector version:

0.1.40

Stripe source connector lookback window:

1 day

Stripe source connector Data request time increment:

1 day

Hi Team, If my source is postgres and destination is s3 and I want to perform custom query on two source tables and save result in single file in s3. Is it achievable? In existing airbyte system I could not see any option if we can write custom query on tables which we can write a result in single csv file. Can someone please help if it is achievable or not.

Hey everyone! I'm curious if anyone has experience bringing in daily spend data from Facebook marketing. I am trying to figure out how much each campaign is cost everyday and I'm having trouble.

Hi All, In S3 destination once data is written.. immediately I want to invoke a rest call in code for Athena table which will create table for written columns. I want to know since it is not springboot application, how can I invoke a rest call in airbyte code base and where can I can write it...

Hello guys, I am trying to build a custom connector but when I run

./gradlew :airbyte-integrations:connectors:source-custom:build

I get an error:

> Task :airbyte-commons:compileJava FAILED

What should I do? I am using java 11.0.17

Hi Team, I have an interesting scenario. New airbyte instance running in EC2 on t3.large instance. Here’re my notes. • Github to Redshift connection • Airbyte v0.40.18, connectors use latest (updated yesterday) • Replication config is the default recommended config • Full Sync took 1hr 20m • Next sync took 1hr 19m I expect next sync (there are few tables with incremental dedupe sync) to be much faster, but it’s not the case. Is there any reason why this is not faster? Thanks.

Hi All. Im trying to use airbyte in Kubernetes (k3s), we have a postgres source and a postgres destination within the same cluster. When I try to do the sync, it fails in the normalization process. (Failure Origin: normalization, Message: Something went wrong during normalization). Im including the log file for this process. I'm just hoping that this issue is not he same as this one: https://github.com/airbytehq/airbyte/issues/5091 because that means we will not be able to use airbyte at all 🫠 Thanks in advance.

In large scale deployment, is docker based deployment recommended or is there any other method?

Guys, is there a way to add a custom field to the syncing table e.g. with the name of my source? Like airbyte adds metadata fields such as _airbyte__ab_id, _airbyte__enitted_at, etc

Hi If I add a aws athena dependency in gradle file, where can I find artifacts or jar of that dependency?

Hi all, has there been anything new on the non-json response issue? I am trying to set up a GCP MySQL source and I keep getting that error. Airbyte is running in GCP compute engine.

Hi team, I have loading JIRA data onto s3 in both parquet and csv format separately but there is a schema mismatch in csv format where many keys(including custom fields) are missing under the ‘fields’ column for ‘issues’ entity. Also while loading data in csv, the quote character is present in unexpected places which is making it difficult for us to create table even with valid csv properties(quote_char, escape_char). Can someone please help here.

Hi guys i have airbyte installed on kubernetes by helm, how can i clean up DB safely so it can run all migration scripts (flyweigh/liquibase) correctly?

Hi Team, Once we setup source and destination and while creating connection we need to select schemas and tables which needs to write in destination. Problem here is if we select 2 tables then in destination it will create two files for each table. Is there any possibility while creating a connection we write a custom query and merge this two table into one file? If not is there any plan for development this functionlity?

Excuse my guys, could you explain me so i understand your perception - who came up with the wonderful idea to release version - 0.41.0 - full of bugs, new features which were eventually rolled back, at friday 18.11, just before the weekend (time window - all devops guys want to use for deployment) ??????

Hi All, I am trying to figure out how to get job Id in s3 destination once we sync the connection. Based on this jobId I need to perform few operations as per our company needs. Can someone please help me how to get current Job Id in s3 destination. I am still trying to figure out how jobs runs and how jobs are mapped with s3 destination.