https://linen.dev logo
#contributing-to-airbyte
Title
# contributing-to-airbyte
c

charles

02/03/2021, 12:02 AM
is anyone already looking into the build issues?
Copy code
AcceptanceTests > initializationError FAILED
    java.lang.IllegalStateException: Check failed: Docker environment should have more than 2GB free disk space
        at org.testcontainers.DockerClientFactory.check(DockerClientFactory.java:298)
        at org.testcontainers.DockerClientFactory.checkDiskSpace(DockerClientFactory.java:287)
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:224)
        at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
        at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
        at io.airbyte.test.acceptance.AcceptanceTests.init(AcceptanceTests.java:143)
u

user

02/03/2021, 12:03 AM
not looking into it right now but my guess is that the images we install during CI has grown too large for Github’s liking
u

user

02/03/2021, 12:04 AM
it is very possible we’re building > 14gb worth of images (I guess 12, since docker needs 2gb free)
u

user

02/03/2021, 12:04 AM
one fix could be to remove all standard source test images for connectors that don’t use them
u

user

02/03/2021, 12:04 AM
(AFAIK only google sheets needs the non-file-based standard tests)
u

user

02/03/2021, 12:05 AM
this commit seems to have pushed it over the edge?
u

user

02/03/2021, 12:05 AM
i'm actually not sure if that makes sense or just happens to be when the disk died.
u

user

02/03/2021, 12:05 AM
i think it’s the latter
u

user

02/03/2021, 12:06 AM
that commit doesn’t touch the core codebase, the main CI build, or anything that gets put into a docker image
u

user

02/03/2021, 12:06 AM
it does though, no?
u

user

02/03/2021, 12:06 AM
Copy code
uses: ridedott/merge-me-action@v1
u

user

02/03/2021, 12:06 AM
Copy code
uses: rtCamp/action-slack-notify@master
u

user

02/03/2021, 12:06 AM
these are newly added docker images that run in the CI, no?
u

user

02/03/2021, 12:07 AM
but those only get run when you run the
publish
command not when you run the main build
u

user

02/03/2021, 12:07 AM
so like, they shouldn’t be getting installed into the
gradle.yml
build environment?
u

user

02/03/2021, 12:08 AM
ah. what you say makes sense.
u

user

02/03/2021, 12:08 AM
also the only net package we’re adding here is
merge-me-action
. the other is already used in
gradle.yml
u

user

02/03/2021, 12:08 AM
dunno. maybe something doesn't work the way we think it does?
u

user

02/03/2021, 12:08 AM
after that commit every subsequent commit on master fails.
u

user

02/03/2021, 12:08 AM
publish workflows from braches are all succeding.
u

user

02/03/2021, 12:10 AM
maybe the cached contents are too large on master?
u

user

02/03/2021, 12:10 AM
so either the master build environments aren't hermetic that machine is filled up or something. or that stuff is geting pulled into the master build env somehow?
u

user

02/03/2021, 12:11 AM
is there an easy way to test what you just mentione jared?
u

user

02/03/2021, 12:11 AM
we could add a step that outputs the size of cached directories
u

user

02/03/2021, 12:11 AM
after it loads them
u

user

02/03/2021, 12:11 AM
or we could nuke by changing
CACHE_VERSION
and check
u

user

02/03/2021, 12:12 AM
i defer to you. you've played with this much more than i have.
u

user

02/03/2021, 12:14 AM
i’ll just nuke for now
u

user

02/03/2021, 12:15 AM
rerunning the master build
u

user

02/03/2021, 12:19 AM
thanks.
u

user

02/03/2021, 1:07 AM
still failing for the same reason
u

user

02/03/2021, 1:07 AM
not sure why
u

user

02/03/2021, 1:16 AM
is the caching separate by workflow? e.g. main build caching separate from that of PR builds?
u

user

02/03/2021, 2:44 AM
yeah
u

user

02/03/2021, 3:02 AM
Thanks.
u

user

02/03/2021, 4:56 PM
what's the next thing to try? just remove all of the caching and see what happens? i can do this if you feel like that's what we should try next.
u

user

02/03/2021, 5:25 PM
Next thing to try is to find out what is taking up all the space in a step in the gradle workflow and output it
u

user

02/03/2021, 5:27 PM
We need to find what is taking up all the space and why that changed recently
u

user

02/03/2021, 5:27 PM
I’m not planning on looking into this today. Need to focus on Temporal stuff.
u

user

02/06/2021, 7:40 AM
I also get this error once in a while locally.., i usualky run
docker system prune
to get out of it but it's annoying...
u

user

02/08/2021, 6:18 PM
I am going to run this experiment to see if it helps: https://github.com/airbytehq/airbyte/pull/1998
u

user

02/08/2021, 6:19 PM
i am choosing this experiment because i could do it in <1 minute of work. The logging filesystem output will take me time to figure out. assuming this does not fix anything (which i think is the expectation) then we will need to dedicate some real time to this.