Channels
advice-data-architecture
advice-data-ingestion
advice-data-orchestration
advice-data-privacy
advice-data-quality
advice-data-transformation
advice-data-visualization
advice-data-warehouses
advice-reverse-etl
airbyte-cloud
airbyte-for-power-users
airbyte-help
airbyte-plus-airflow
airbyte-plus-dagster
announcements
authentication
careers
cloud-master-build-failure
community-strategy
connector-builds
connector-development
contributing-to-airbyte
databricks-airbyte316
databricks-airbyte316
delete-remove
dev
dev-3-alerts
dev-db-sources
dev-frontend
dev-frontend6323
dev1-alerts
events-and-conferences
ext-torch-cube
feedback-and-requests
github-updates
good-reads
gtm
hacktoberfest-2022
help
infra-dev-alerts
infra-dev-alerts-webhook
introductions
issue
kubernetes
license-questions
linen-test
local-brazil
local-london
local-sf
marcos-test-discourse
memes-and-banter
mos-client
mos-client
motleyfool
movedata-a-different-way-to-work
movedata-airbyte-connection-management-scale
movedata-airbyte-orchestration-in-gcp
movedata-better-data-testing-with-the-data-error-generating-process
movedata-bring-your-own-infra
movedata-building-a-real-time-user-facing-dashboard-with-airbyte-redpanda-and
movedata-building-connectors-in-minutes
movedata-cicd-for-data-building-devtest-data-environments-with-lakefs
movedata-data-analysts-are-setup-to-fail-and-its-our-fault
movedata-data-engineering-is-software-engineering-and-software-engineering-is-da
movedata-data-orchestration-is-not-just-running-jobs-on-a-schedule
movedata-dataops-on-the-open-modern-data-stack
movedata-dev-first-open-source-data-observability-solution
movedata-ducky-data-crunching-on-the-laptop---the-pendulum-swings
movedata-five-causes-of-data-quality-issues
movedata-future-evolution-of-the-data-stack
movedata-general
movedata-guardrails-not-stop-signs
movedata-if-you-build-it-will-they-come-how-to-activate-your-modern-data-stack
movedata-in-2023-data-trust-will-be-more-important-than-ever-before
movedata-ingesting-data-with-airbyte-into-a-high-speed-data-lakehouse-using-trin
movedata-keynote-building-the-foundations-of-data-movement
movedata-let-your-data-team-choose-their-own-tools
movedata-mobilize-the-worlds-data
movedata-modern-data-management---how-to-achieve-data-discovery
movedata-moving-data-reliably-ingestion-observability-are-better-together
movedata-navigating-the-modern-data-stack-with-open-source-headless-bi
movedata-open-source-communities-shape-modern-data-stacks
movedata-prefect-ing-self-hosted-airbyte
movedata-prep-your-pipelines---reverse-etl-and-the-coming-great-flood
movedata-speakers
movedata-streaming-made-modern
movedata-the-best-data-warehouse-is-a-lakehouse
movedata-the-data-ecosystem-is-ready-for-etl-to-be-dead
movedata-the-end-of-the-pipeline
movedata-traditional-data-catalogs-will-be-replace-by-active-metadata-platforms
movedata-using-airbyte-on-day-1of-our-startup
movedata-using-airbyte-to-build-your-dremio-open-data-lakehouse
movedata-who-needs-a-traditional-data-warehouse-when-you-can-get-real-time-analy
movedata-why-bi-is-not-enough
octavia-cli
octavia-life
office-hour
office-hour-12-october
oss-master-build-failure
p0-cloud-syncs-failing-dec-15
p0-instagram-incompatible-metrics
p0-source-airtable-stream-failures
partner-productpair-hightouch
prod1-alerts-test
prod1-health
random-news-and-events
shameless-plugs
troubleshooting
understanding-airbyte
using-the-cdk
write-for-the-community
Title
c
Chris (deprecated profile)
10/23/2020, 4:57 PMHello, what is the task/process that generates
airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py./gradlew buildroot:rootu
user
10/23/2020, 4:58 PMfrom
airbyte-integrations/bases/base-python/build.gradlegenerateProtocolClassFilesDoes that mean your gradle is running as root somehow?
no
can you add
whoamigenerate-protocol-files.sh./gradlew generateProtocolClassFiles(maybe it's the docker daemon ran by root though?)
docker isn't creating this file though
oh nvm
completely forgot
maybe we need to add
--user "$(id -u):$(id -g)"are you also getting everything created as root (logs, etc) if you run an integration through the airbyte app?
the
whoamii guess yes?
ls -l /tmp/airbyte_local/output_data/exchange_rate.csv
-rw-r--r-- 1 root root 13447 Oct 23 19:11 /tmp/airbyte_local/output_data/exchange_rate.csvif you add
--user "$(id -u):$(id -g)"generate-protocol-files.shyes that's much better
@charles do you think we should add this user scoping everywhere?
in my opinion, definitely (for dev users not on mac os...) and the Dockerfiles should probably include the recommended snippet from your linked post too then:
ARG USER_ID
ARG GROUP_ID
RUN addgroup --gid $GROUP_ID user
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID user
USER userBTW this is also why your build failed here:
https://github.com/airbytehq/airbyte/pull/684/checks?check_run_id=1299333311
Task :airbyte-integrations:bases:base-python:blackFormat FAILED
[python] .venv/bin/python -m black . --line-length 140
error: cannot format /home/runner/work/airbyte/airbyte/airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py: [Errno 13] Permission denied: '/home/runner/work/airbyte/airbyte/airbyte-integrations/bases/base-python/airbyte_protocol/models/airbyte_message.py'why did that just start happening though?
because it never caused problems until blackFormat needs to rewrite the generated python file and can't because it's owned by root
ahh
good point
I'm going to try adding this to the generation
does that snippet replace haing to do --user everwhere?
no you should do both
no it forces you to set user and group everywherre
^^
I guess maybe we don't care as much for everything in-app?
code tooling seems highest priority
but it does seem reasonable that people want to run airbyte as a user, just not as high priority
actually this is specifying at build time
which is a bit strange
I don't think we actually want that since people will often be pulling these images
ah
generateProtocolClassFilesit wouldn't be too bad to add these fixes at least to the docker build/run for
generate-protocol-files.shI got the build passing by explicitly excluding regeneration in the later gradle runs
So it's merged now
What I'm most worried about the fixes is the fact that it ties UIDs/GIDs to the artifact (the image) which seems strange.
It's fine for something like code generation but it doesn't feel right for anything we'd publish?
for the code generation code, we provide the actual ID when calling the
docker runtrue. for code generation especially it probably doesn't need more than the --user flag