Hello, so I am modifying the spark-lineage plugin to try some improvements but I keep getting this error which I don't really know the source of it:

Exception in thread "map-output-dispatcher-3" java.lang.UnsatisfiedLinkError: com.github.luben.zstd.Zstd.setCompressionLevel(JI)I

I am building the jar with the following command

./gradlew metadata-integration:java:spark-lineage:buildDependents

, and the source code I have started from is the code of v0.9.1 tag. The modifications done do not modify yet the base functionality, they are just more log prints. The application submitted worked with the v0.9.2 of the spark-lineage plugin, so I am guessing that this error could have been some step that I have unconsciously skipped in the build process. (I skipped the tests as I was stuck with another error and as it was a checking of the docker deployment, I didn't gave it much importance. Could that be it??)

hmm, which version of docker do you have? Is it possible it is not running? Based on this error message it seems like TestContainers unable to find docker environment

I double checked in case it wasn't, but docker was indeed running. I will read the github issue in case it applies, thank you Tamas.

Odd, can you run docker containers manually via the cli? Perhaps the user doesn’t have permission to use docker? Does the hello world test for that user work?

docker run hello-world

Oh my bad, when I installed the VM I thought I gave the user root privileges but it seems like I was mistaken. Changing the user I was able to build it correctly, thank you!!

What happens if you run your spark job without our jar?

It prints the same error, although at first it didn't. Could it be possible that the instalation of my compiled jar could updated or modified something of the package mentioned (zstd-jni)?

it shouldn’t, you can try to remove our jar

And hdfs?? I imagine that it doesn't modify it either but it could be (as the jar is moved to HDFS after the download process if I am not mistaken)