Hi team. I try to build the project based on commit

commit 451aa97bfa6406883ca59fb5632d3e4f9cf2cb72 (HEAD -> master, origin/master, origin/HEAD)
Author: Charles <giardina.charles@gmail.com>
Date:   Fri Apr 16 17:13:08 2021 -0700

Java14

siamakhaschemi@localhost airbyte % java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)

Python 3.7.9 managed by pyenv

siamakhaschemi@localhost airbyte % pyenv version
3.7.9 (set by /Users/siamakhaschemi/Development/ing/airbyte/.python-version)

virtualenv

siamakhaschemi@localhost airbyte % virtualenv --version
13.1.2

When I run gradlew on the top-level of the project, I get an error for pythonenv

siamakhaschemi@localhost airbyte % ./gradlew clean build --stacktrace
[...]
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
> Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':airbyte-integrations:bases:airbyte-protocol:checkPython'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:200)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:198)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:179)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
[...]
Caused by: org.gradle.api.GradleException: Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (<https://github.com/xvik/gradle-use-python-plugin/issues>). You can disable virtualenv usage with 'python.scope = USER'.
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.checkPython(CheckPythonTask.groovy:82)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.switchEnvironment(CheckPythonTask.groovy:156)
        at ru.vyarus.gradle.plugin.python.task.CheckPythonTask.run(CheckPythonTask.groovy:61)
        at jdk.internal.reflect.GeneratedMethodAccessor1043.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$3.run(ExecuteActionsTaskExecuter.java:555)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
[...]
Caused by: org.gradle.process.internal.ExecException: A problem occurred starting process 'command '.venv/bin/python''
        at org.gradle.process.internal.DefaultExecHandle.execExceptionFor(DefaultExecHandle.java:241)
        at org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultExecHandle.java:218)
        at org.gradle.process.internal.DefaultExecHandle.failed(DefaultExecHandle.java:369)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:87)
        at org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
        ... 3 more
Caused by: net.rubygrapefruit.platform.NativeException: Could not start '.venv/bin/python'
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
        at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
        at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:98)
        at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:71)
        ... 4 more
Caused by: java.io.IOException: Cannot run program ".venv/bin/python" (in directory "/Users/siamakhaschemi/Development/ing/airbyte/airbyte-integrations/bases/airbyte-protocol"): error=2, No such file or directory
        at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
        ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
        ... 8 more

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv
lib

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib
python3.9

siamakhaschemi@localhost airbyte % ls airbyte-integrations/bases/airbyte-protocol/.venv/lib/python3.9
config-3.9-darwin lib-dynload

Seems that the virtualenv is not created in a way that it contains a python binary. Do I need to install

pipenv

also? I am new to python, maybe it’s a trivial problem.

Hello, I have started using Airbyte to integrate social media data and I like it so far. I'm not sure about how Airbyte workers scale ? We may have multiple social media connections per user and a lot of users in the near future, is it safe to use Airbyte on GCP as described in the doc to handle it ?

@Jared Rhizor (Airbyte) or @charles did you guys have to do anything to help intellij pick up the

postgresl.conf

so

CdcPostgresSourceStandardTest

can be run in Intellij?

Hi I am building Airbyte using " ./gradlew build " command but it gave me below error "OracleSourceTest > testReadSuccess() FAILED java.sql.SQLException: Cannot create PoolableConnectionFactory (ORA-00604: error occurred at recursive SQL level 1 ORA-01882: timezone region not found" Any solution to resolve this error ?

Hi guys, do you have any idea why this

[35mairbyte-server      |[0m 2021-04-20 19:25:00 [1;31mERROR[m i.a.s.RequestLogger(filter):105 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.20.0.1 POST 500 /api/v1/source_definitions/list_latest - {}

could happen? got 500 from

source_definitions/list_latest

and where should I look for logs?

Hello! I’m trying to hook up integration tests in

base-normalization

project (using a

customIntegrationTestPython

task) However, in this test, I actually need to run some docker images of destinations, how do i make my task depends on them in the build.gradle? (i’ve seen in the connectors side, they successfully depends on normalization airbyte docker images but i can’t do it the way around?) I get:

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/chris/Workspace/airbyte/airbyte-integrations/bases/base-normalization/build.gradle' line: 14

* What went wrong:
A problem occurred evaluating project ':airbyte-integrations:bases:base-normalization'.
> Could not get unknown property 'airbyteDocker' for project ':airbyte-integrations:connectors:destination-bigquery' of type org.gradle.api.Project.

Hey everyone, been playing around and so far I love it. Is there an endpoint to get all workspaces? Or a way to get the uuid for the default workspace? Can the ui be visited just for an individual workspace?

Is anyone experiencing issue with

:airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFilesWithoutLicense

hanging? Was about to merge my change to the styling stuff but then ran into this issue on master.

Hello! 👋 First of all, thanks to everyone working on Airbyte...I just discovered it, and it has been really awesome so far! I have a question about a Shopify integration, in particular, but the question applies more generally to how load and transform happen. I am trying to load Shopify orders + order_refunds + transactions into our BigQuery instance, and I was hoping to get the orders into one table (i.e. not normalized); however, when I turn off the ‘normalization’ option in the UI, I only seem to be able to see the JSON blob in _airbyte_data. Is there some middle-ground option where the data can be spread into columns in BigQuery, but not completely normalized into separate tables? I did notice this documentation here, which was interesting: https://docs.airbyte.io/tutorials/connecting-el-with-t-using-dbt Is the ‘transformation’ step something that we, as the consumer, are responsible for? It would be awesome if the data could fit into one table, given that BigQuery supports the RECORD type, so the nesting works pretty well in that destination

Sorry I'm some how confused,  When i have defined spec.json , also there is config.json. But all the specs in spec.json are provided by the user in the UI. Will it be automatically saved to config.json?. Because it is the one parsed in implementation, i some saved manually the configuration then where are the configuration provided in the UI saved?

Hi all. After spending some time here in the weekend i finally build my first connector (source). Super good experience and i am excited to see the future of airbyte 🙂 However is there any particular reason why it is not possible to do scd on full_refresh tables? I mean in some cases it might be necessary to track hard deletes or maybe the source do not have a way to do incremental extract? Or is this determined by the destination?

We’re already hitting the vCPU limit for EC2 requested build workers (@Davin Chia (Airbyte) fyi)

anyone else having trouble compiling from a clean install (make sure that your pyenv is starting from scratch, not just

./gradlew clean

)? failing on task

:airbyte-integrations:bases:base-normalization:installLocalReqs

. it's a bunch of C level errors when trying to compile pyarrow. I feel like I've seen this before and the answer was just to downgrade the version of the python library we were using. I know something in normalization got upgraded again. Being kinda lazy and asking this question, because last time I dug into this, I lost like a day and a half of my time.

[itt]

airbyte-cli

build failure

is anyone about to run

./gradlew :airbyte-integrations:bases:airbyte-protocol:generateProtocolClassFilesWithoutLicense

successfully?

reminder to the team (and everyone in this channel!) to update your macs for the new security patch: https://www.theverge.com/2021/4/26/22404124/mac-macos-update-11-3-security-ios-app-m1-gaming-controller-music

is there a way to run all integration tests?

Hi! Im working on creating a new connector and doing so in Visual Studio Code. Im not getting any syntax help with the airbyte stuff. Has anybody got that working? Im getting this error in VS Code:

Unable to import 'airbyte_protocol'pylint(import-error)

Today, one of the most annoying parts of my workflow is remembering to run

./gradlew format

before pushing a change. Do other people feel the same way? How would people feel if Github did this for us?

Hello everyone! I want to build custom connectors but don't know how to start with it. Any reference for the same?

Anyone also see this behavior on the docs pages where links for

incremental - append

and

incremental - deduped

are not clickable? https://www.loom.com/share/cbf95776e602475c8a563b7aa7e30edf

As per API doc, connection creation doesn't ask

workspaceId

, (https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create) while, there is a

workspaceId

required for fetching list of connections? (https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/list) How the default is picked up for workspaceId in API of connection create? Is there any update on connection create API which has included workspaceId.

while doing a release, I’m running into

4531e200ac8d: Layer already exists
ed3fe3f2b59f: Layer already exists
b2d5eeeaba3a: Layer already exists
0.23.0-alpha: digest: sha256:b0cd277cd03b0a90a7ffecda65ada9a1364918a2d48dcc41830ed18e1cc4cbca size: 1986
Pushing migration (airbyte/migration:0.23.0-alpha)...
The push refers to repository [<http://docker.io/airbyte/migration|docker.io/airbyte/migration>]
ERROR: An image does not exist locally with the tag: airbyte/migration

howdy all - been playing with Airbyte for a couple of days now. Trying to get a feel for it and how it would best serve our needs. I tried to deploy it to K8s on Minikube (Linux Mint) but got stuck at a few steps, primarily with the fact that it was pinned to an old version and that I couldn't seem to make it work without having a deployment for temporal running along side the other resources (for which there wasn't a shared resource as far as I could gather). After a couple tweaks etc. I've got it working. I was wondering whether it's worth adding this to a PR, perhaps updating the docs on K8s support too. Any guide to where to get started would be great 🙏

which source connectors (python based) have cursor support? Looking for code examples...

good morning! I’m building a data pipeline using argo events / argo workflows / argocd, because they’re great with k8s. was wondering if anyone on the team has experience integrating these tools with airbyte for orchestration

does anyone know why we dont use ORDER BY clause in our select query here ?

i'm using localstack for development rather than aws services — has anyone attempted to source files from localstack's s3 rather than The Real S3 in airbyte (for instance by changing the endpoint url or using importing localstack as boto3)

Hi! Im new to Airbyte. My aim is to create a connector to a system with a REST API. This API has for some objects lists of related objects, ie hierarchy. My first approach was to break these lists out as bridge tables inside the python connector code. Now I am wondering if that maybe is not the best approach. I think I read somewhere that there is a way to create dbt-models inside Airbyte or integrate the two. Maybe replicating the data just as is in Airbyte and then do the flattening of the data in dbt instead of in the Airbyte connector. Is my plan sound and if so how do I integrate Airbyte and dbt?

@charles or anyone, I have a specific question around

JdbcStreamingQueryConfiguration

. I see that all the implementations of the interface are doing this

@Override
  public void accept(Connection connection, PreparedStatement preparedStatement) throws SQLException {
    connection.setAutoCommit(false);
    preparedStatement.setFetchSize(1000);
  }

My question is why do we need to set the

AutoCommit

as false if we are just using a SELECT query to fetch the data?