Heads up Airbyte team, I've figured out a short term fix for the build failures we've been seeing. TLDR: • The test/publish flows will also run on ec2 instances on AMI images that contain all the dependencies required to build Airbyte. See https://github.com/airbytehq/airbyte/pull/4558/files for more context. Lmk if anyone sees issues/has questions! • The frontend builds will also run on the ec2 runner • In short we are fully on the ec2 runners now - should also see a mild boost in speeds!

@Davin Chia (Airbyte) Is

virtualenv

the only thing that is missing ? I tried installing it but exception remains. I don't build it inside the docker though. I run gradle build from my host system which is Ubuntu 21.04. Any suggestions ? Here is partial call-stack:

FAILURE: Build failed with an exception.

* Where:
Build file '/home/expert/work/sideprojects/airbyte/airbyte-cdk/python/build.gradle' line: 2

* What went wrong:
An exception occurred applying plugin request [id: 'airbyte-python']
> Failed to apply plugin 'airbyte-python'.
   > java.lang.ClassNotFoundException: precompiled_AirbytePython

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.plugins.InvalidPluginException: An exception occurred applying plugin request [id: 'airbyte-python']
        at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.exceptionOccurred(DefaultPluginRequestApplicator.java:207)
        at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.applyPlugin(DefaultPluginRequestApplicator.java:189)
        at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.lambda$applyPlugins$1(DefaultPluginRequestApplicator.java:127)
        at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.applyPlugins(DefaultPluginRequestApplicator.java:127)
        at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.applyPlugins(DefaultPluginRequestApplicator.java:123)
        at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:117)
        at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:56)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$run$1(DefaultBuildOperationExecutor.java:74)

hmm that's a different stacktrace than the one I was seeing though it seems to be related to the same plugin

yes can you try and manually install

virtualenv

before running

./gradlew build

once more? something like

python3 -m pip install virtualenv==16.7.9 --user

Yes, I did exactly that (from your PR) but it didn't change the error. Too bad stacktrace doesn't give much information. What is

airbyte-python

resposible for? Here is the most inner exception. Does it give you any ideas?

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: precompiled_AirbytePython
        at AirbytePythonPlugin.apply(AirbytePythonPlugin.java:23)
        at AirbytePythonPlugin.apply(AirbytePythonPlugin.java:10)
        at org.gradle.api.internal.plugins.ImperativeOnlyPluginTarget.applyImperative(ImperativeOnlyPluginTarget.java:43)
        at org.gradle.api.internal.plugins.RuleBasedPluginTarget.applyImperative(RuleBasedPluginTarget.java:51)
        at org.gradle.api.internal.plugins.DefaultPluginManager.addPlugin(DefaultPluginManager.java:187)
        at org.gradle.api.internal.plugins.DefaultPluginManager.access$100(DefaultPluginManager.java:52)
        at org.gradle.api.internal.plugins.DefaultPluginManager$AddPluginBuildOperation.run(DefaultPluginManager.java:282)

No it doesn't. This makes me think there is some Gradle corruption or Java runtime mismatch.

what Java and Python versions are you on?

Java 16, Gradle 7.1 (I fixed few problems in your script to make it work) and Python 3.9.5

that might be it. we are on Java 14, Gradle 6 and Python 3.7

in the spirit of cyclomatic complexity I would minimise the change set and try again 😛