Hi all, is anyone else building datahub behind a c...
# contribute-code
a
Hi all, is anyone else building datahub behind a corporate firewall with restricted access to external repositories (maven, nodejs, ...)? If so, what's your setup? I'm considering opening a PR to make all repositories configurable via GRADLE_OPTS. Would this help anyone else besides my team? I'm thinking something along the lines of (repositories.gradle)
```repositories {
gradlePluginPortal()
mavenLocal()
if (project.hasProperty('apacheMavenRepositoryUrl')) {
maven { url project.getProperty('apacheMavenRepositoryUrl') }
} else {
mavenCentral()
}
if (project.hasProperty('confluentMavenRepositoryUrl')) {
maven {
url project.getProperty('confluentMavenRepositoryUrl')
}
} else {
maven {
url "https://packages.confluent.io/maven/"
}
}
if (project.hasProperty('linkedinOpenSourceRepositoryUrl')) {
maven {
url project.getProperty('linkedinOpenSourceRepositoryUrl')
}
} else {
maven {
url "https://linkedin.jfrog.io/artifactory/open-source/" // GMA, pegasus
}
}
}```
as well as (build.gradle in various subprojects)
docker {
...
// Add build args if they are defined (needed for some CI or enterprise environments)
def dockerBuildArgs = [:]
if (project.hasProperty('alpineApkRepositoryUrl')) {
dockerBuildArgs.ALPINE_REPO_URL = project.getProperty('alpineApkRepositoryUrl')
}
if (project.hasProperty('openTelemetryDownloadUrl')) {
dockerBuildArgs.OPEN_TELEMETRY_REPO_URL = project.getProperty('openTelemetryDownloadUrl')
}
if (project.hasProperty('mavenCentralRepositoryUrl')) {
dockerBuildArgs.MAVEN_CENTRAL_REPO_URL = project.getProperty('mavenCentralRepositoryUrl')
}
if (dockerBuildArgs.size() > 0) {
buildArgs(dockerBuildArgs)
}
}
+ required changes to the dockerfiles:
```FROM alpine:3 AS base
# Configurable repositories
ARG ALPINE_REPO_URL=http://dl-cdn.alpinelinux.org/alpine
ARG OPEN_TELEMETRY_REPO_URL=https://github.com/open-telemetry/opentelemetry-java-instrumentation
ARG MAVEN_CENTRAL_REPO_URL=https://repo1.maven.org/maven2
# Optionally set corporate mirror for apk
RUN if [ "${ALPINE_REPO_URL}" != "http://dl-cdn.alpinelinux.org/alpine" ] ; then sed -i "s#http.*://dl-cdn.alpinelinux.org/alpine#${ALPINE_REPO_URL}#g" /etc/apk/repositories ; fi
# Upgrade Alpine and base packages
# PFP-260: Upgrade Sqlite to >=3.28.0-r0 to fix https://security.snyk.io/vuln/SNYK-ALPINE39-SQLITE-449762
RUN apk --no-cache --update-cache --available upgrade \
&& apk --no-cache add curl sqlite \
&& apk --no-cache add openjdk11-jre --repository=${ALPINE_REPO_URL}/edge/community \
&& apk --no-cache add jattach --repository ${ALPINE_REPO_URL}/edge/community/
ENV JMX_VERSION=0.18.0
RUN wget ${OPEN_TELEMETRY_REPO_URL}/releases/download/v1.24.0/opentelemetry-javaagent.jar -O opentelemetry-javaagent.jar \
&& wget ${MAVEN_CENTRAL_REPO_URL}/io/prometheus/jmx/jmx_prometheus_javaagent/${JMX_VERSION}/jmx_prometheus_javaagent-${JMX_VERSION}.jar -O jmx_prometheus_javaagent.jar```
What does everyone think of this? It would allow us to build datahub by providing a bunch of custom jfrog/artifactory urls for these values.
plus1 1
Opened a PR - can someone from the team please take a look if I missed anything? 🙂 Thanks! https://github.com/datahub-project/datahub/pull/9230
r
Hey - thanks for raising the PR! cc. @brainy-tent-14503 as a reminder to review this :)
❤️ 1
a
Thanks! Also, let me know if you would want to see mention of these changes somewhere in the docs or so.
This has been a bit stale unfortunately. Should I be more patient, or is there anything I can do to help kick off the review / discussion? 🙂 Happy to hop on a call or something as well.
r
Just haven't had the bandwidth to review this yet. Looks good to me, I'll resolve the conflicts and then get it merged once the CI is green. Thank you very much!
a
ah, sorry. Didn't mean to be pushy! Thank you!
r
No worries, I definitely appreciate the PR!