Title
c

Chris (deprecated profile)

10/23/2020, 4:57 PM
Hello, what is the task/process that generates
airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py
? For some reason, when i run my
./gradlew build
it ends up with a new ownership by
root:root
user and makes my command fails afterward because of permission access... ⁉️
u

user

10/23/2020, 4:58 PM
from
airbyte-integrations/bases/base-python/build.gradle
generateProtocolClassFiles
Does that mean your gradle is running as root somehow?
no
can you add
whoami
in
generate-protocol-files.sh
and run
./gradlew generateProtocolClassFiles
(maybe it's the docker daemon ran by root though?)
docker isn't creating this file though
oh nvm
completely forgot
maybe we need to add
--user "$(id -u):$(id -g)"
to the docker call?
are you also getting everything created as root (logs, etc) if you run an integration through the airbyte app?
the
whoami
returns the user not root in the script shell
i guess yes?
ls -l /tmp/airbyte_local/output_data/exchange_rate.csv 
-rw-r--r-- 1 root root 13447 Oct 23 19:11 /tmp/airbyte_local/output_data/exchange_rate.csv
if you add
--user "$(id -u):$(id -g)"
to just the docker call in
generate-protocol-files.sh
does it fix your problem?
yes that's much better
@charles do you think we should add this user scoping everywhere?
in my opinion, definitely (for dev users not on mac os...) and the Dockerfiles should probably include the recommended snippet from your linked post too then:
ARG USER_ID
ARG GROUP_ID

RUN addgroup --gid $GROUP_ID user
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID user
USER user
BTW this is also why your build failed here: https://github.com/airbytehq/airbyte/pull/684/checks?check_run_id=1299333311
Task :airbyte-integrations:bases:base-python:blackFormat FAILED
[python] .venv/bin/python -m black . --line-length 140
	 error: cannot format /home/runner/work/airbyte/airbyte/airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py: [Errno 13] Permission denied: '/home/runner/work/airbyte/airbyte/airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py'
why did that just start happening though?
because it never caused problems until blackFormat needs to rewrite the generated python file and can't because it's owned by root
ahh
good point
I'm going to try adding this to the generation
does that snippet replace haing to do --user everwhere?
no you should do both
no it forces you to set user and group everywherre
^^
I guess maybe we don't care as much for everything in-app?
code tooling seems highest priority
but it does seem reasonable that people want to run airbyte as a user, just not as high priority
actually this is specifying at build time
which is a bit strange
I don't think we actually want that since people will often be pulling these images
ah
generateProtocolClassFiles
is working the first time but not the second
it wouldn't be too bad to add these fixes at least to the docker build/run for
generate-protocol-files.sh
, right?
I got the build passing by explicitly excluding regeneration in the later gradle runs
So it's merged now
What I'm most worried about the fixes is the fact that it ties UIDs/GIDs to the artifact (the image) which seems strange.
It's fine for something like code generation but it doesn't feel right for anything we'd publish?
for the code generation code, we provide the actual ID when calling the
docker run
not building the image
true. for code generation especially it probably doesn't need more than the --user flag