I'm writing a custom source connector. It is an REST API. I'm having issues with the schema. The problem is that this is a "custom field" so the attributes are rarely consistent between one object to the next. I would like it to leave it as an json object in the destination. Can I just put the type as "{"type": ["null", "object"]}" and will it still work?

Hi! I’m trying to run the source and destination containers without the airbyte server, i.e., running the containers by hand has shown in the docs. The source one I’m using (Google Ads) is fine, but once I pipe the json output to the postgres destination container, it builds the table, but it’s empty. Is this expected behavior?

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hello airbyte. I am giving your product a first spin using the web interface. Data from a tiny schema transfers without errors, but a schema with more tables/data gets the following error: "Additional Failure Information: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!" I am doing a Postgres->Postgres replication. I have tried overwriting the destination data, appending, and incremental without dups. All get the same error. The source is an on-prem db and the destination is in AWS. It sort of seems the data has transferred. I have not done exhaustive tests. What might this indicate? Fwiw my schema has about 300 tables, all identical in structure. My logs are attached.

Ehy guys, here a typical state

{
  "campaign_analytics": {
    "DATE": "2021-06-09"
  }
}

My question is, would it be possible to have a nested state?

{
  "campaign_analytics": {
    "campaign_1": {
      "DATE": "2021-06-09"
    },
    "campaign_2": {
      "DATE": "2020-06-09"
    }
  }
}

Unfortunately all the examples use the deprecated method

get_updated_state

So is it possibile to have a nested state? If so, do you have an example how to use it?

Hi Team, if I set up a source with a parameter of type array, how do I need to add it in the UI? I am getting this error

Anyone have an example of how to add GOOGLE_APPLICATION_CREDENTIALS into my airbyte docker? I have tried adding it as a volume and them specifying the env variable to a few services but it never finds it.

@here, I am going to try Airbyte deployment on EKS. (not going to use plural) Any do/ don't for me?

Hi ,I'm currently using airbyte (version 39.37)for ingesting table from postgresql (version 13) to mysql(version 5.6) when we select sync mode for tables replication Out of 3 sync mode (full refresh append ,full refresh Overwrite,and increment append)only Full refresh append and full refresh overwrite are available in some tabels. as we have set increment append for all table But its not seen .Kindly Help me to Resolve issue.

hi - how to increase the fetch size of the sync? its just pulling 10k rows per pull. for 10m rows table literally taking 3 hrs to complete

Hi Folks, I just started using Airbyte, my source is Clickhouse and my destination is GCS, I want to store the data in GCS from a clickhouse table to different folders for e.g, different users data in different path in GCS, Is it possible?

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

hi, is anyone able to sync large tables(like 10 M rows table to snowflake/Azure/GCS) from local. If yes, what would the right configuration. I am trying for past 2 days not able to move forward. I am using Postgres -> Snowflake(Via AWS S3 bucket stage) its running for 3 hours and failing at some point but nothing is loading but i can see the files in AWS S3 bucket. Below is my .env file

# Relevant to scaling.
SYNC_JOB_MAX_ATTEMPTS=3
SYNC_JOB_MAX_TIMEOUT_DAYS=3
JOB_MAIN_CONTAINER_CPU_REQUEST=8
JOB_MAIN_CONTAINER_CPU_LIMIT=8
JOB_MAIN_CONTAINER_MEMORY_REQUEST= 100g
JOB_MAIN_CONTAINER_MEMORY_LIMIT= 100g


### LOGGING/MONITORING/TRACKING ###
TRACKING_STRATEGY=segment
JOB_ERROR_REPORTING_STRATEGY=logging
# Although not present as an env var, expected by Log4J configuration.
LOG_LEVEL=INFO


### APPLICATIONS ###
# Worker #
# Relevant to scaling.
MAX_SYNC_WORKERS=10
MAX_SPEC_WORKERS=10
MAX_CHECK_WORKERS=10
MAX_DISCOVER_WORKERS=10
# Temporal Activity configuration
ACTIVITY_MAX_ATTEMPT=
ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS=
ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS=
WORKFLOW_FAILURE_RESTART_DELAY_SECONDS=

I received an email about Airbyte changing its pricing from compute time-based to volume-based with no acknowledgement or reasoning as to why you switched. The original pricing page made the case for why Airbyte didn’t like volume-base pricing, so I’m wondering if there’ll be a post or other explanation for this switch. Thanks!

Hi team, i tried to create a java destination connector . while following a step mentioned in https://docs.airbyte.com/connector-development/tutorials/building-a-java-destination/ link . i ran "./generate.sh" command in gitbash and it is throwing *"While trying to generate a connector, an error occurred on line 38 of generate.sh and the process aborted early. This is probably a bug.*" error. that paricular line no is actually having "docker run --rm -it --name airbyte-connector-bootstrap --u0ser "$_UID:$_GID" -e HOME=/tmp -v "$(pwd)/../../../.":/airbyte airbyte/connector-bootstrap" command.

Anyone who could hep me on airbyte helm charts deployment?

I am trying to clone this airbyte repo and getting error Filename is too long. Please find the attached snapshot

Hi all, I'm just getting into airbyte and am currently trying to mirror some tables from a mysql database source to a postgresql destination. That works great so far, however the data types are lost. In particular mysql DATETIME columns are converted to strings and i need them to be of type TIMESTAMP in postgresql. What would be the best way to achieve this?

I am working through deploying Airbyte to openshift using the helm charts provided. I am using version 0.33.1 but have had to patch a number of broken things that I see are in the queue to being reviewed and fixed. I have however the following problem which I am struggling getting past. The temporal container fails with this issue: 'Unable to start server. Error: TLS provider initialization error: invalid ServerTLS.ClientCAData: CA cannot be empty string'. I am using external postgres and external minio. Can any one give me a hit as to what feature has TLS enabled and needs to be turned off?

Hello. I have developed and tested an Azure storage option for the Databricks connector. However, as part of this, I refactored the existing Databricks connector which previously only supported S3 storage. I do not have S3 credentials so I cannot test the S3 option locally before submitting the PR. Will the acceptance tests be run automatically with Airbyte’s S3 credentials when I open the PR?

how to disable all the table sync and select required. As we have more than 500 tables here on top of that i just need 2 tables to load.. May be i am missing some thing here

Hi Team How can we use Google Analytics v4 as a source in Airbyte Cloud? We have tried to setup GA, but we get errors like

The connection tests failed.

"Invalid custom reports json structure. ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=75497580, lib=9, reason=108, reason_text=b'error0480006CPEM routines::no start line')])"

When we check the source code, we see 2 sources ( source-google-analytics-data-api and source-google-analytics-v4). But in Airbyte cloud we have only 1 choice.

Does the Octavia CLI only look in

~/.octavia

for

FOO=bar

to use in sources, destinations, and connections via

${FOO}

?

hi.. just would like to understand if we have 10 tables to sync. does Airbyte goes one by one or all in parallel?.. In my case I see 1 large tables running and others not yet started.. Is there any place where we increase this parallelism?..

HI everyone, is anyone using the google analytics connector and pulling in conversions? I can't figure out what table that information is coming in on.

Hello Everyone, Good evening. I have been experimenting self hosted airbyte for my organisation and came with couple of questions or feature requests. Could someone help me with the below? 1. Since we wanted to make our ETL system more robust and self-service oriented, I’m looking something like : Having multiple workspace for different teams such as marketing, controlling, pricing and so on. Each team can make their different sources but designated target, Which means pricing team can connect sources like airtable, gsheets, postgres but defaulting to snowflake target database pricing_db. Similarly Marketing team can connect wide variety sources but target to marketing db. This saves lot of effort in data governance and data consolidation and so on. Do we have something features like that available? 2. Is there a way that I can setup RBAC as like Analysts who can just see connections and synch status, developer who can create sources, destinations and connections and so on. 3. Is there a generic gitlab ci file / repo with self-service oriented? Meaning users would be creating connections only via yml file and apply would happen with octavia cli. Would be great incase someone could help us with the above requirement!z Thanks in advance!

Hi. I am new to this community.. I am doing Airbyte POC for my company.. If anyone interested to learn together, please DM, we will connect via ZOOM on daily basis and explore Airbyte 🙂

GUYS need help; I have a connection with Redshift as destination: despite I clearly specify the destination schema, the connection keeps inserting data into public 😢 any idea why this is happening?

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hi Everyone, I am currently using the airbyte for bringing in the monday.com data into our warehouse. Currently the Airbyte is asking to provide Access Token, Client ID and Client Secret. I have the token with me but I am not able to find the other 2. Can anyone help me with this issue? Thanks