Siamak Haschemi
04/17/2021, 11:48 AMcommit 451aa97bfa6406883ca59fb5632d3e4f9cf2cb72 (HEAD -> master, origin/master, origin/HEAD)
Author: Charles <giardina.charles@gmail.com>
Date: Fri Apr 16 17:13:08 2021 -0700
Java14
siamakhaschemi@localhost airbyte % java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
Python 3.7.9 managed by pyenv
siamakhaschemi@localhost airbyte % pyenv version
3.7.9 (set by /Users/siamakhaschemi/Development/ing/airbyte/.python-version)
virtualenv
siamakhaschemi@localhost airbyte % virtualenv --version
13.1.2
When I run gradlew on the top-level of the project, I get an error for pythonenv
siamakhaschemi@localhost airbyte % ./gradlew clean build --stacktrace
[...]
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:200)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:198)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:179)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
[...]
Caused by: org.gradle.api.GradleException: Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.
at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.checkPython(CheckPythonTask.groovy:82)
at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.switchEnvironment(CheckPythonTask.groovy:156)
at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.run(CheckPythonTask.groovy:61)
at jdk.internal.reflect.GeneratedMethodAccessor1043.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:555)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
[...]
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command '.venv/bin/python''
at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
... 3 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start '.venv/bin/python'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
... 4 more
Caused by: java.io.IOException: Cannot run program ".venv/bin/python" (in directory "/Users/siamakhaschemi/Development/ing/airbyte/airbyte-integrations/bases/airbyte-protocol"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv
lib
siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib
python3.9
siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib/python3.9
config-3.9-darwin lib-dynload
Seems that the virtualenv is not created in a way that it contains a python binary.
Do I need to install pipenv
also?
I am new to python, maybe it’s a trivial problem.Maxime Naulleau
04/19/2021, 9:36 AMDavin Chia (Airbyte)
postgresl.conf
so CdcPostgresSourceStandardTest
can be run in Intellij?Ankur Adhikari
04/20/2021, 3:11 PMEugene Kulak
[35mairbyte-server |[0m 2021-04-20 19:25:00 [1;31mERROR[m i.a.s.RequestLogger(filter):105 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.20.0.1 POST 500 /api/v1/source_definitions/list_latest - {}
could happen?
got 500 from source_definitions/list_latest
and where should I look for logs?Chris (deprecated profile)
base-normalization
project (using a customIntegrationTestPython
task)
However, in this test, I actually need to run some docker images of destinations, how do i make my task depends on them in the build.gradle?
(i’ve seen in the connectors side, they successfully depends on normalization airbyte docker images but i can’t do it the way around?)
I get:
FAILURE: Build failed with an exception.
* Where:
Build file '/Users/chris/Workspace/airbyte/airbyte-integrations/bases/base-normalization/build.gradle' line: 14
* What went wrong:
A problem occurred evaluating project ':airbyte-integrations:bases:base-normalization'.
> Could not get unknown property 'airbyteDocker' for project ':airbyte-integrations:connectors:destination-bigquery' of type org.gradle.api.Project.
Wesley Gormley
04/22/2021, 9:12 PMcharles
:airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFilesWithoutLicense
hanging? Was about to merge my change to the styling stuff but then ran into this issue on master.Charles McMillan
04/23/2021, 8:16 PMCorrensa
04/23/2021, 8:57 PMFrederik Hagelund
04/25/2021, 4:10 PMJared Rhizor (Airbyte)
charles
./gradlew clean
)? failing on task :airbyte-integrations:bases:base-normalization:installLocalReqs
. it's a bunch of C level errors when trying to compile pyarrow. I feel like I've seen this before and the answer was just to downgrade the version of the python library we were using. I know something in normalization got upgraded again. Being kinda lazy and asking this question, because last time I dug into this, I lost like a day and a half of my time.Jared Rhizor (Airbyte)
airbyte-cli
build failureDavin Chia (Airbyte)
./gradlew :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFilesWithoutLicense
successfully?Davin Chia (Airbyte)
Vitor Avancini
04/28/2021, 12:53 PMMartin Larsson
04/28/2021, 7:53 PMUnable to import 'airbyte_protocol'pylint(import-error)
Davin Chia (Airbyte)
./gradlew format
before pushing a change. Do other people feel the same way? How would people feel if Github did this for us?Jay Patel
05/03/2021, 7:15 AMChris (deprecated profile)
incremental - append
and incremental - deduped
are not clickable?
https://www.loom.com/share/cbf95776e602475c8a563b7aa7e30edfSandeep Dange
05/03/2021, 3:10 PMworkspaceId
, (https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create)
while, there is a workspaceId
required for fetching list of connections? (https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/list)
How the default is picked up for workspaceId in API of connection create?
Is there any update on connection create API which has included workspaceId.Jared Rhizor (Airbyte)
4531e200ac8d: Layer already exists
ed3fe3f2b59f: Layer already exists
b2d5eeeaba3a: Layer already exists
0.23.0-alpha: digest: sha256:b0cd277cd03b0a90a7ffecda65ada9a1364918a2d48dcc41830ed18e1cc4cbca size: 1986
Pushing migration (airbyte/migration:0.23.0-alpha)...
The push refers to repository [<http://docker.io/airbyte/migration|docker.io/airbyte/migration>]
ERROR: An image does not exist locally with the tag: airbyte/migration
Coetzee van Staden
05/04/2021, 6:41 PMAndrew Padilla
05/06/2021, 3:05 PMbradfair
05/06/2021, 3:34 PMSubodh (Airbyte)
bradfair
05/07/2021, 1:44 PMMartin Larsson
05/09/2021, 3:14 PMSubodh (Airbyte)
JdbcStreamingQueryConfiguration
. I see that all the implementations of the interface are doing this
@Override
public void accept(Connection connection, PreparedStatement preparedStatement) throws SQLException {
connection.setAutoCommit(false);
preparedStatement.setFetchSize(1000);
}
My question is why do we need to set the AutoCommit
as false if we are just using a SELECT query to fetch the data?