Howdy, is it intentional that the docker images fo...
# all-things-deployments
silly-apple-97303
09/03/2020, 8:10 PMHowdy, is it intentional that the docker images for the consumer jobs are using alpine linux?
https://github.com/linkedin/datahub/blob/master/docker/datahub-mce-consumer/Dockerfile#L4
Specifically running into errors like this because alpine linux does not use glibc. I think this error is only showing up in our real kafka environments/not docker compose because our kafka cluster is configured to use snappy compression by default.
Copy code
java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-bb847a5e-21b5-4d9b-babd-f31afc7109a7-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/snappy-1.1.7-bb847a5e-21b5-4d9b-babd-f31afc7109a7-libsnappyjava.so)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)s
steep-airplane-62865
09/03/2020, 8:21 PMThat was an intentional change from
openjdk:8openjdk:8-jre-alpinesteep-airplane-62865
09/03/2020, 8:23 PMI think you can always build your custom images by forking it as a solution but before going into that route, it's not clear to me why you need
glibcs
silly-apple-97303
09/03/2020, 8:26 PMWe have snappy compression enabled on all of our kafka topics by default.
It looks like the kafka consumer is using a snappy native library to do the decompression from the topic, and that native library is linked against glibc
silly-apple-97303
09/03/2020, 8:28 PMThere might be a way to install a compat layer for libc into alpine as described here:
https://github.com/gliderlabs/docker-alpine/issues/219
I'll let you know if it works.
silly-apple-97303
09/03/2020, 9:28 PMI think that libc6-compat layer worked, still working through some other issues once I got past the snappy library loading issues.
Any issues if I open a PR adding that for the consumers (after some more testing)?
s
steep-airplane-62865
09/03/2020, 9:33 PMThanks for working that out. We always appreciate contribution 🙂 Waiting for your PR.
s
silly-apple-97303
09/04/2020, 8:53 PMDid not forget about this, just trying to get schema registry/ssl sorted out so I can do an end to end test.
Will prob open a PR early next week.
👍 1
silly-apple-97303
09/28/2020, 9:29 PMFinally opened a PR for this, not sure if this is the best approach I gave a possible alternative in the PR:
https://github.com/linkedin/datahub/pull/1899/files