https://linen.dev logo
Join Slack
Powered by
I am using Ubuntu 18.04 for the development enviro...
# contributing-to-airbyte
p
I am using Ubuntu 18.04 for the development environment. I have satisfy all requirement mention in Doc. My first build is fail like so:
Copy code
> Task :airbyte-integrations:connectors:destination-jdbc:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:destination-jdbc:test'.
> There were failing tests. See the report at: file:///root/airbyte/airbyte-integrations/connectors/destination-jdbc/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 32m 3s
428 actionable tasks: 266 executed, 49 from cache, 113 up-to-date
Gradle was unable to watch the file system for changes. The inotify watches limit is too l
u
Copy code
root@airbyte:~/airbyte# java --version
java 14.0.2 2020-07-14
Java(TM) SE Runtime Environment (build 14.0.2+12-46)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

root@airbyte:~/airbyte# node --version
v14.17.0

root@airbyte:~/airbyte# python --version
Python 3.8.10

root@airbyte:~/airbyte# docker --version
Docker version 20.10.7, build f0df350

root@airbyte:~/airbyte# docker-compose --version
docker-compose version 1.29.2, build 5becea4c

root@airbyte:~/airbyte# jq --version
jq-1.5-1-a5b5cbe

root@airbyte:~/airbyte# cmake --version
cmake version 3.10.2

root@airbyte:~/airbyte# psql --version
psql (PostgreSQL) 10.17 (Ubuntu 10.17-0ubuntu0.18.04.1)
u
Hi, can you run that specific task with 
--stacktrace
and 
--info
?e.g. 
./gradlew :airbyte-integrations:connectors:destination-jdbc:test  --stacktrace --info
u
and post where the specific error is happening?
u
Because of docker disk space?
Copy code
to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
    2021-06-09 04:28:44 INFO o.t.DockerClientFactory(client):210 - {} - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
    2021-06-09 04:28:44 INFO o.t.DockerClientFactory(client):221 - {} - Checking the system...
    2021-06-09 04:28:44 INFO o.t.DockerClientFactory(check):295 - {} - ✔︎ Docker server version should be at least 1.6.0lsTest
    2021-06-09 04:28:45 ERROR o.t.DockerClientFactory(check):297 - {} - ❌ Docker environment should have more than 2GB free disk space

SqlOperationsUtilsTest > testInsertRawRecordsInSingleQuery() FAILED
    java.lang.IllegalStateException: Check failed: Docker environment should have more than 2GB free disk space
        at org.testcontainers.DockerClientFactory.check(DockerClientFactory.java:298)
        at org.testcontainers.DockerClientFactory.checkDiskSpace(DockerClientFactory.java:287)
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:224)
        at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
        at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
        at io.airbyte.integrations.destination.jdbc.SqlOperationsUtilsTest.setup(SqlOperationsUtilsTest.java:64)

SwitchingDestinationTest > testCopy() STANDARD_OUT
    2021-06-09 04:28:48 INFO i.a.i.d.j.c.SwitchingDestination(getConsumer):83 - {} - Using destination type: COPYtionTest
    2021-06-09 04:28:48 INFO i.a.i.d.j.c.SwitchingDestination(check):75 - {} - Using destination type: COPY

SwitchingDestinationTest > testInsert() STANDARD_OUT
    2021-06-09 04:28:48 INFO i.a.i.d.j.c.SwitchingDestination(getConsumer):83 - {} - Using destination type: INSERTonTest
    2021-06-09 04:28:48 INFO i.a.i.d.j.c.SwitchingDestination(check):75 - {} - Using destination type: INSERT

Gradle Test Executor 1 finished executing tests.

> Task :airbyte-integrations:connectors:destination-jdbc:test FAILED
u
that seems right
u
try restart the docker agent with more disk space?
u
Yes you correct, issue solved. Thanks
u
Other fail, I had virtualenv install properly.
Copy code
> Task :airbyte-integrations:connectors:source-amazon-seller-partner:unitTest FAILED
:airbyte-integrations:connectors:source-amazon-seller-partner:unitTest (Thread[Daemon worker Thread 3,5,main]) completed. Took 2.153 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:source-amazon-seller-partner:unitTest'.
> Python call failed: .venv/bin/python -m pytest -s unit_tests
u
can you do the same as above and post the gradle stack trace?
u
that result i send is from stack trace
u
is that the only python task that is failing?
u
there isn't a better error on why python isn't running well?
u
yes for now it is the only task that fail
u
what are you trying to do? if you are working on something other than the amazon-seller-partner source, this failure can be ignored
u
you can run just the specific task you want to work on
u
I want to work on S3 Destination Connector, would like to check the possibility of modify and will contribute back to community if useful.
u
Basically I want to work on this issue
u
u
nice!
u
as long as you can run 
./gradlew :airbyte-integration:connectors:destination-s3:build
you should be good
u
It was success with that specific component and core. My postgreSQL service working fine but Airbyte seem like could not connect? I got this after using: VERSION=dev docker-compose up
Copy code
airbyte-scheduler   | Host db:5432 not yet available...
airbyte-server      | Host db:5432 not yet available...
airbyte-temporal    | waiting for postgresql to start up
airbyte-temporal    | nc: bad address 'db'
airbyte-temporal    | + '[' 1 -eq 0 ']'
airbyte-temporal    | + echo 'waiting for postgresql to start up'
airbyte-temporal    | + sleep 1
airbyte-temporal    | + nc -z db 5432
u
what does 
docker stats
output?
u
Copy code
CONTAINER ID   NAME                CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O    PIDS
11647c344b3f   airbyte-temporal    0.02%     1008KiB / 3.852GiB    0.02%     1.29kB / 252B   0B / 0B      2
7453cd3003e4   airbyte-webapp      0.00%     2.016MiB / 3.852GiB   0.05%     1.54kB / 0B     0B / 4.1kB   2
fc54d9721f5d   airbyte-scheduler   0.02%     844KiB / 3.852GiB     0.02%     586B / 0B       0B / 0B      2
f61ebce509ba   airbyte-server      0.03%     820KiB / 3.852GiB     0.02%     516B / 0B       0B / 0B      2
u
looks like your DB isn't up
u
postgres in host is working fine, but you correct airbyte-db is not seen here
u
that should start with 
docker-compose up
u
can you run 
docker-compose down -v
and then 
docker-compose up
and monitor the docker containers to see why Postgres isn't starting up?
u
no need for 
VERSION=dev
since you aren't developing on the Core Airbyte systems
u
similarly you don't need to build core airbyte when working on the S3 connector. running 
./gradlew :airbyte-integration:connectors:destination-s3:build
builds a dev image. if you set the S3 destination connector version to 
dev
in the admin page, the latest 
dev
image will always be used each time
u
@Davin Chia (Airbyte) Yes thank you that's work. But after i make change on code I always have to stop docker-compose and rebuild the component to see the effect. Is there any faster way? any suggestion?
u
are you sure?
u
you shouldn't need to do that as long as you build the image and set the version to 
dev
. the next run should automatically pick it up
u
i set dev in connector admin page only, That is not enough?
u
that should be enough
u
similarly you don't need to build core airbyte when working on the S3 connector. running 
./gradlew :airbyte-integration:connectors:destination-s3:build
 builds a dev image. if you set the S3 destination connector version to 
dev
 in the admin page, the latest 
dev
 image will always be used each time
I follow your suggestion, but hmm it not automatically pickup the change. I working on this directory 
/airbyte-integrations/connectors/destination-s3/src/main
u
it's late my side now - but I'm happy to schedule a call tmrw and pair with you on this
u
feel free to pcik something from https://calendly.com/davin-airbyte/30min-1
3 Views
Open in Slack
PreviousNext
Powered by