Hi there. We’re using Airbyte to load data into snowflake. Currently we have more than 50+ connections configured and working. In one connection - from mysql to snowflake we have problem - initial sync failed and we got error

Stack Trace: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Buffer length is less than expected payload length.
	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:67)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
	at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$0(IntegrationRunner.java:187)
	at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:237)

and more

"failureOrigin" : "source",
  "failureType" : "system_error",
  "internalMessage" : "java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Buffer length is less than expected payload length.",
  "externalMessage" : "Something went wrong in the connector. See the logs for more details.",

and

"failureOrigin" : "source",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!",
  "externalMessage" : "Something went wrong within the source connector",

and

"failureOrigin" : "replication",
  "internalMessage" : "java.lang.RuntimeException: java.io.UncheckedIOException: java.io.IOException: Stream closed",
  "externalMessage" : "Something went wrong during replication",

after that I see data dbt process, but tables are empty. Any suggestions how to fix it?

hello team, My source is mysql and the destination is postgres is there any workaround to copy artifacts (constraints like pkey and fkey) from source to destination?

In the source when we're defining the schema with limited fields and additionalProperties being True still its failing the test case for the connector. even though we have mentioned 'additionalProperties': True still its not updating when the connector is doing basic read

Failed validating 'additionalProperties' in schema:
    {'$schema': '<http://json-schema.org/draft-08/schema#>',
     'additionalProperties': False,
     'properties': {'_id': {'type': 'integer'},
                    'endtime': {'format': 'date-time', 'type': 'string'}},
     'type': 'object'}

Hi! We are testing the kafka source and we found that it's not using the schema registry integration to create an AirbyteCatalog - it returns a dummy schema. I haven't found any issue about this on github

Hi - could I, pls, get some assistance with my question https://airbytehq.slack.com/archives/C021JANJ6TY/p1681315393161049?thread_ts=1680170930.009079&cid=C021JANJ6TY ?

Hi team, I’m trying to import the data from amazon seller partner and fetching the data for

GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL

table and I always get no records for some strange reason. I have tried by all sync modes, transformation. I even did everything from scratch with all different combinations but I always get no records and there is no errors also. It gets weird, I have another connection running for same amazon seller partner and there I’m getting data for

GET_LEDGER_SUMMARY_VIEW_DATA

and

GET_FBA_INVENTORY_PLANNING_DATA

and they are working all good. I don’t understand why I’m getting no data for

GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL

. Any help is very much appreciated. Thanks. 🎉

Hello, can I create a source connection to an FTP using port 21?, you only have to connect to SFTP, even changing it generates an error.

Hi team, How to create source definition for all workspace? I need Rest API details to create source definition for all workspace. 1. Im using Airbyte open source. 2. Airbyte Version - 0.42.1 3. Im trying to add the custom connector from UI. But it only created current workspace. unable to view it from another workspace.

heyaa !! has anybody created a salesforce connection , need some help on how to get the "refreshtoken"

Hi , I am new to Airbyte. I have Airbyte with Kafka as source and Snowflake is destination with sync mode is append. Kafka message is in Avro format. But sync performance is very slow. Even for 40k messages it took 1hour. Should I add any special settings or schema registry for Avro? What would be the schema for Avro format kafka message ?Can anyone suggest?

Hi team, why am I getting no records for

GET_FLAT_FILE_ALL_ORDERS_DATA_BY_LAST_UPDATE_GENERAL

in amazon seller partner connection, while other data imports are working fine.

Hi Team

hii what destination can I use for free account

This message was deleted.

Hi i am in the last step for deploy airbyte on gcp

gcloud --project=$PROJECT_ID beta compute ssh $INSTANCE_NAME -- -L 8000:localhost:8000 -N -f

at this step "_create an SSH tunnel to connect the GCP instance to Airbyte"_ in my local terminal i have this error message "unknow option "-f" some have a clue on this ?

Does Octavia-CLI have any capability to Delete an Airbyte connection/source? What is the best way to go about it if we are using Octavia-CLI for performing CRUD operations on our Airbyte resources.

Hello Airbyte team, We have a couple of connectors whose spec sizes exceed the current maximum size of 32KB. Is it possible to make this limit configurable? We’ve filed a GitHub issue for this but we can also contribute the change if necessary.

Hi #C021JANJ6TY, I am using Airbyte with source-kafka and destination-snowflake. Does anyone knows If Sync failed or canceled , what happens kafka messages? Will I loose them

Hi #C021JANJ6TY, I am new to Airbyte. I have Airbyte with source-Kafka and destination-Snowflake with sync mode is incremental-append & earliest offsetrest. Kafka message is Avro format. But sync performance is very slow. Even for 40k messages it took 1hour. Should I add any special settings or schema registry for Avro? What would be the schema for Avro format kafka message ?Can anyone suggest?

Hi #C021JANJ6TY, I am new to Airbyte and going to deploy the open source Airbyte on Mac. Can I just follow the steps in the documentation below to set up?https://docs.airbyte.com/deploying-airbyte/local-deployment/

Hi! how can I know if airbyte current version is compatible with k8s 1.21 ?

Hii can you provide detail list of transformation that we can do on data

Is there a setting I need to toggle to get a job to fail if it finds a schema validation error on the source side?

Hi, I am trying to extend pipedrive source connector by adding DealFlows stream, this endpoint doesn't support a cursor field so I cannot pull it incrementally, but it's parent stream 'Deals' can be synced incrementally so I want to pull it's parent stream incrementally and pull the substream for only the deal_ids that are updated so it can be incremental in theory. But I cannot access parent stream's state from substream so I cannot pull the parent incrementally. Is there a workaround for this? And why am I not able to pull parent stream incrementally? Any help is appreciated.

Hi, I am trying to add a data validation stage to our data pipeline using great expectations. Please, I haven't found any resources that detail how this can be done. Our current Airbyte moves data from Postgres to Google Bigquery. Question: Is this integration possible? Also, any guide on how to go about this?

I have like 2 tokens I need to use for authentication. How can I pass in 2 tokens and 2 headers into the config?

Hi All, I have installed Airbyte open source on a kubernetes cluster in AWS. I want to enable single sign on gmail authentication on the airbyte instance . Can you please tell me what are the steps of doing this?

Hi - I’m running Airbyte on GKS using the helm chart. Whenever I run a connection check it fails; the newly created pod to facilitate the check doesn’t get created with the

airbyte-admin

service account created as part of the helm chart and instead just uses the

default

as per this screenshot. Is this a known issue? How can I work around this? Do I need to provide the relevant IAM grants to the default account in the namespace to access my external resources etc.?

hii team please let me know google analytics will be considered as database source or api source and also provide me detail list of api sources

Hi team, How to create new connector for all workspaces in Airbyte Open Source? Airbyte Version - 0.42.1