@gunu @Harshith (Airbyte) the fix for https://github.com/airbytehq/airbyte/issues/4378 has been implemented and released as 0.27.2-alpha. Give it a shot and lmk if you still see errors happening!

Hi Team, Is there any plan to incorporate spark in airbyte?

When I try to run the template generators locally they seem to hang forever. Do these run on other people's machines? Trying to figure out if it's something up with my environment. command:

./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates --scan

result:

...

<-------------> 0% EXECUTING [2m 17s]
> :airbyte-integrations:connector-templates:generator:testScaffoldTemplate_Java JDBC Source

Heads up! I just merged a change that restructures are builds. The goals of this change are: 1. to allow people working on connectors or platform to work faster and more autonomously (e.g. avoid a connector build failure blocking a platform release). 2. allow us to optimize the builds separately. they scale differently and we want to make sure we are being efficient where possible. This changes the commands you will use to build the project.

CORE_ONLY

is gone. The new syntax is documented here. Let me know if you have any issues or questions! Changes like this are bound to have some rough edges, so please don't hesitate to speak up if something does not look right.

Hi everyone, not sure if this channel is the right place to put that message but I’d love to have your thoughts on a topic. How do you manage to reconcile data objects from multiple data sources? example: • Link/associate an invoice on quickbooks to a Deal on Hubspot • Link/associate a stripe subscription to an Account on Salesforce • Link a task on Asana to a google calendar event • Link a Github issue or pull request to a Jira ticket • Link objects between business tools (not necessarily CRMs) I’d love to discuss with anyone who though about connecting/associating all the different data objects that live inside SaaS tools

Here wanted to get some understanding on these points - related to k8s deployment 1. Is there any restriction on max syncs that can happen at any point of time. a. We were facing issues like

address is already in use

b. The sync becomes stale and no movement in logs after waiting for pod with ip 2. I understand that we have some ports defined which the jobs can use is there a way increasing the ports will help in increasing total number of syncs. 3. When pod moves to completed state can we release the ports? 4. How many ports does 1 sync job takes ?

Guys, not sure when the problem was introduced, but I can’t develop UI in the last master - I am trying to call

localhost:8000/api

route from

localhost:3000

and get CORS errors which I was not receiving before… If I open localhost:8000 - webapp is opened. So I suppose something was changed in CORS or routing

Hi, We are facing similar problems when transporting a large HubSpot into BigQuery (https://github.com/airbytehq/airbyte/issues/4779). The task fails at the normalization stage, giving "Bad double value" when attempting compile .sql files. When generating sql, it return WARN: Unknown type datetime for column. I also suspect the issue is with the sql. Thanks everyone for building Airbyte!

how is intellij doing for you with our monorepo? i've noticed mine start to chug alot.

accidentally forgot to add 1 env variable to docker-compose file. Here is the fix for the build https://github.com/airbytehq/airbyte/pull/4844

@Davin Chia (Airbyte) is the recommended way to add a CLI to the AWS runners to add it to the AMI? I want to add the

aws

cli to the build runner

Hi, I’m trying to connect my local Airbyte to a custom job / config database that is running on my localhost, but when I try to start airbyte with docker-compose up, it keepks showing the message : “waiting for postgresql to start up”. Basically what I did was to change the database variables in the .env file, as follows, thanks!

DATABASE_USER=postgres
DATABASE_PASSWORD=my_local_pwd
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_DB=airbyte
DATABASE_URL=jdbc:<postgresql://localhost:5432/airbyte>

Hello, I am setting up airbyte in a k8s cluster. I tried importing a configuration from my local setup. After doing that, the sources, destinations, and connection pages will not load. I can see requests to

<http://localhost:8000/api/v1/sources/list>

are returning a 500. Is there a way to clear the imported configuration?

Hello small and lame question. For a connection where do we store the state ?

trying to call workspace/list on 27.0 and get next error

Access to fetch at '<http://localhost:8001/api/v1/workspaces/list>' from origin '<http://localhost:3000>' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: It does not have HTTP ok status.

All other endpoints are working correctly.

In our current project, we need to group the settings ( connections ) by “users” or “groups”, for example, my customer A needs to have their connections and customer B needs to have its own connections, so, one does not see each other. I’m planning to do some work in my current fork to accomplish this, I just want to know if is there any already implemented solution for this case, thanks!

Hey folks! Is there any chance a new deployment of the airflow-cdk could be published? It looks like some of the python destination features are available in the main repo but are in version 0.1.6 which is not on pypi yet Edit: Whoops, I see now there's a release candidate, does anyone know what the timeline for release might be?

Huge shoutout to @Mario Molina for contributing the newly released Kafka destination! 🎉 Mario was a sterling example of how collaboration between people in the OSS community leads to a high quality outcome. If you’re happily moving data to Kafka, make sure to drop him a thank you!

Hello, the current master is broken 😞?

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-commons:compileJava'.
> Could not resolve all files for configuration ':airbyte-commons:compileClasspath'.
   > Could not resolve com.therealvan:appender-log4j2:3.1.2-SNAPSHOT.
     Required by:
         project :airbyte-commons
      > Could not resolve com.therealvan:appender-log4j2:3.1.2-SNAPSHOT.
         > Unable to load Maven meta-data from <gcs://airbyte-test-public-bucket/com/therealvan/appender-log4j2/3.1.2-SNAPSHOT/maven-metadata.xml>.
            > Could not get resource '<gcs://airbyte-test-public-bucket/com/therealvan/appender-log4j2/3.1.2-SNAPSHOT/maven-metadata.xml>'.
               > Failed to get Google credentials for GCS connection

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>

Is it me or are Automatic Migration Acceptance Test flakey? I feel like I keep seeing transient failures where it is complaining about connection to the db.

Any BigQuery SQL user up to solve a challenge? 🎁 How do you cleanly extract a json column from this example? 1. I have one solution that is not so acceptable because it modifies the json value data… 2. the other twos are throwing errors from BigQuery • But in ideal, I would want a column named:

column_'with"_quotes

(which may not be possible in BQ) • or maybe I might be ok with another name such as

column__with__quotes

• and my row should still have the value of:

How to extract this json "column_'with"_quotes" in BigQuery?

with example_json_data as (
  select
    "{'column_\'with\"_quotes': 'How to extract this json \"column_\'with\"_quotes\" in BigQuery?'}" as nested_json
)
select 
  json_extract(
      replace(`nested_json`, "column_'with\"_quotes", "column__with__quotes"), "$['column__with__quotes']" -- but i don't want to modify the data value!
      --`nested_json`, '$."column_\'with\"_quotes"' -- throws error: Invalid token in JSONPath at: ."column_'with"_quotes"
      --`nested_json`, "$['column_\'with\"_quotes']" -- throws error: Invalid token in JSONPath at: ['column_'with"_quotes']
    ) as column__with__quotes
from example_json_data

Hi team i found error "WORKER_ENVIRONMENT not found or empty, defaulting to DOCKER" with multipal sources like mailchimp, slack, facebook. Anyone have any idea about this error.

attempting to get a local build of airbyte working

👋 Hello, team!

FYI disabling push to master on our repo

In

StandardSync

, should

namespaceDefinition

be a required field?

I want to edit the notification content, how should I handle it? please help me! 😞

This message was deleted.

could you explain me how to do this?

Put up an RFC PR with a proposal for handling the 

final

 keyword in java. Would like to hear what the rest of the team thinks. Please feel free to respond in the PR. If we are going to move forward with it, I'll shoot to clean up and merge on Monday.