Master build is failing due to an error in `format`:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-cdk:python:checkPython'.

https://github.com/airbytehq/airbyte/runs/2667103448 However, I cannot reproduce this locally. Is anyone able to reproduce it?

For the Kubernetes work, we’re planning on changing the internal usage of

ProcessBuilderFactory

to something that returns a

Process

instead of a

ProcessBuilder

, so more of a

ProcessFactory

. This is because we don’t to allow the various

ProcessBuilder

modifications that aren’t supported across both Docker and Kubernetes. This implementation is just a minor class refactor. Just wanted to get a gut check if anyone feels like this is a terrible idea.

Would it make sense for us to move the upgrading airbyte tutorial to the top-level of the docs? A really common problem we have is that people try to upgrade without them discovering the upgrade docs.

Hey @Davin Chia (Airbyte), If I am not wrong, this gradle configuration for MyPy: https://github.com/airbytehq/airbyte/blob/c61fc3df072aba97e264b715e0096f9d3605abc6/buildSrc/src/main/groovy/airbyte-python.gradle#L99 It is not working, right? that’s why you made this instead?

airbyte-cdk/python/type_check_and_test.sh

: https://github.com/airbytehq/airbyte/blob/c61fc3df072aba97e264b715e0096f9d3605abc6/airbyte-cdk/python/type_check_and_test.sh#L3

@Chris (deprecated profile) do you have a rough sense for how long the

source-file

integration test should run for?

https://github.com/airbytehq/airbyte/pull/3704 I’m out of the loop on normalization, trying to get context on this PR. Are we still keeping de-nesting long term?

Regarding normalization and https://github.com/airbytehq/airbyte/issues/1927, I know that a general solution might be far away, but I'm wondering if we could get to structured rows by hand-editing a DBT model to do the appropriate transformation for our specific tables. Is that something you think would be worth trying?

I keep seeing this error while trying to build. Do we have a ticket about tracking down and fixing what's causing a build tool to crash?

> Task :spotlessJava FAILED

Step 'removeUnusedImports' found problem in 'whatever file':

null

java.lang.reflect.InvocationTargetException

...big stack trace here

Sometimes I receive error

{
  "message": "The received object did not pass validation",
  "details": "property: getSourceDefinition.arg0.sourceDefinitionId message: must not be null invalid value: null"
}

for the request

*Request URL:* <http://localhost:8001/api/v1/source_definitions/get|http://localhost:8001/api/v1/source_definitions/get>

airbyte-server      | 2021-06-01 17:56:08 ERROR i.a.s.RequestLogger(filter):105 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.26.0.1 POST 400 /api/v1/source_definitions/get - {}

Has anybody faced same issue? I am using hubspot -> pg

@Jared Rhizor (Airbyte) @charles is there a reason we are setting

source_defined_cursor

to true in the Postgres Source's discovery method? my understanding is that users can select any column to be the cursor

@[DEPRECATED] Marcos Marx (and support DRIs in general): For most Kube issues (unless there’s a very clear problem) we should be saying to hold tight for a couple weeks since we’re going to offer much better support that behaves very differently under the hood. We won’t be working towards fixing specific issues that are likely going to be irrelevant after switching to the newer version.

did someone turn on reviewable?!?

Heads Up: Jared and I are noticing some caching weirdness with

composeBuild

, where changes are not getting pulled in. I'm going to create a ticket for this. in the mean time, please run composeBuild with the

--rerun-tasks

flag

Is anyone else seeing intermittent CI failures due to missing python virtualenv stuff? It seems to go away with a rerun, but that just makes it all the more frustrating!

Hello, i have changed port for webapp 8000 to 8080 in .env file and trying to restart my webapp container, but it gives me same error for 8000 port:

ERROR: for webapp  Cannot start service webapp: driver failed programming external connectivity on endpoint airbyte-webapp (756d2638f52f217d3212e3d2b1386ffb7ac29d34bd2390a4bbcdcbd879244585): Bind for 0.0.0.0:8000 failed: port is already allocated

This message was deleted.

heads up: builds are failing because something is funky with our AWS instances. investigating

Any reference reagarding basic normalization in MySQL destination?

Hey guys, I have this MongoDB collection with thousands of docs, which have an id field in them in the form : "_id": { "$binary": "YkrYedM4QKGkq0lkcTZ4pQ==", "$type": "3" }, This is being parsed by airbyte in the string form "#<BSON:Binary0x00007f70d8381350>" How do I configure airbyte to parse it in the form "a8c5b73f-e801-46e0-b722-45b167ac576c" @Aditya Guru

Hey guys, when i tried to configure Redshift destinantion, it gives me error like:

I am following this https://docs.airbyte.io/contributing-to-airbyte/developing-locally to setting up the locally development environment. But got fail during build. Any suggestion? I have install all technologies are required to build Airbyte locally mention at the beginning of the doc already. I am using Window 10 to do this.

PS D:\Self-Project\PPMC\airbyte> ./gradlew clean build
Building all of Airbyte.
D:\Self-Project\PPMC\airbyte\airbyte-integrations\connectors
> Task :airbyte-cli:airbyteDocker FAILED
...
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-cli:airbyteDocker'.
> A problem occurred starting process 'command 'D:\Self-Project\PPMC\airbyte\tools\bin\build_image.sh''

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 5m 16s
159 actionable tasks: 53 executed, 2 from cache, 104 up-to-date

@Davin Chia (Airbyte) Hi this is the output.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD SUCCESSFUL in 13s
37 actionable tasks: 2 executed, 35 up-to-date

We have got a scenario here. We have been pulling data for facebook ads for last 1 year and we realised we want more attributes but we don't want to loose our existing data how do we go about this?

I’m trying to build the core project for the first time, on OSX 10.14.6. I have openjdk 14, and I think brew gave me python 3.9 when I upgraded other libraries. I also have conda python 3.8 available and in my user PATH as my preferred python. However, gradle isn’t finding a usable python during the

checkPython

step, and it’s unclear how to control which python it tries to use. It seems unable to see my conda python at all (despite it being in my path), and when it uses the brew python, virtualenv chokes with the following error.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:base-normalization:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

Is there a way I can control what python is used for building? I’m sure I’m not the only one with a messy python situation and a preference to control which one it finds and tries to use. If this is a version issue, I might be able to pin brew’s version to 3.7 or 3.8 instead of 3.9 (hopefully).

I am using Ubuntu 18.04 for the development environment. I have satisfy all requirement mention in Doc. My first build is fail like so:

> Task :airbyte-integrations:connectors:destination-jdbc:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:destination-jdbc:test'.
> There were failing tests. See the report at: file:///root/airbyte/airbyte-integrations/connectors/destination-jdbc/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 32m 3s
428 actionable tasks: 266 executed, 49 from cache, 113 up-to-date
Gradle was unable to watch the file system for changes. The inotify watches limit is too l

Is there any support for role base access? can we create users?

when updating the destination namespace using the backend API (using the connectionUpdate endpoint), is there a way to use existing

syncCatalog

for the given connection e.g.

${CONNECTION_SCHEMA_CATALOG}

{
    "connectionId": uuid,
    "namespaceDefinition": "customformat",
    "namespaceFormat": "new_schema",
    "status": "active",
    "syncCatalog": ${CONNECTION_SYNC_CATALOG} # instead of pasting a very very long dictionary
}

Guys, what is the best way to manage python versions? Default version for

python

command in macos is 2.7, however according the docs:

We assume `python` points to python >=3.7.

as a result build is failing ( however it was always built fine before ). What is a proper way to make it to be built with python3? • just create alias in .zshrc • pyenv • some other tool?

Does anyone know why some of the connectors (specifically I'm using Google AdWords and Facebook Marketing) just show

Waiting for source thread to join.

in the log output, and then nothing until an error occurs? It seems like the log was intended to show progress output from the connector, and then something caused the output of the subthread to get redirected to the wrong place. What's the expected/desired behaviour?

I fail some of the integration test locally because

java.nio.file.NoSuchFileException: secrets/config.json

where should I specify and what content to be put? I running test on

:airbyte-integrations:connectors:destination-s3:integrationTest