https://datahubproject.io logo
Join Slack
Powered by
I'm getting this error during `./gradlew build` `:...
# getting-started
b
I'm getting this error during 
./gradlew build
:metadata-service:auth-ranger-impl:test
Upon further inspection of the test I see this 🧵
Copy code
java.lang.UnsatisfiedLinkError: /Users/benmarte/Library/Caches/JNA/temp/jna911027812998872738.tmp: dlopen(/Users/benmarte/Library/Caches/JNA/temp/jna911027812998872738.tmp, 0x0001): tried: '/Users/benmarte/Library/Caches/JNA/temp/jna911027812998872738.tmp' (fat file, but missing compatible architecture (have (i386,x86_64), need (arm64e)))
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
	at java.lang.Runtime.load0(Runtime.java:811)
	at java.lang.System.load(System.java:1088)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1012)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988)
	at com.sun.jna.Native.<clinit>(Native.java:195)
	at com.kstruct.gethostname4j.Hostname$UnixCLibrary.<clinit>(Hostname.java:12)
	at com.kstruct.gethostname4j.Hostname.getHostname(Hostname.java:30)
	at org.apache.ranger.plugin.util.RangerRESTUtils.<clinit>(RangerRESTUtils.java:93)
	at org.apache.ranger.plugin.audit.RangerDefaultAuditHandler.<init>(RangerDefaultAuditHandler.java:52)
	at com.datahub.authorization.ranger.DataHubRangerClient.init(DataHubRangerClient.java:39)
	at com.datahub.authorization.ranger.RangerAuthorizer.init(RangerAuthorizer.java:40)
	at com.datahub.authorization.ranger.RangerAuthorizerTest.setupTest(RangerAuthorizerTest.java:99)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61)
	at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366)
	at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320)
	at org.testng.internal.TestInvoker.runConfigMethods(TestInvoker.java:701)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:527)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.runTests(TestNGTestClassProcessor.java:141)
	at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.stop(TestNGTestClassProcessor.java:90)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.lang.Thread.run(Thread.java:748)
I know this is most likely the culprit: 
(fat file, but missing compatible architecture (have (i386,x86_64)
but IDK how to resolve this, any ideas? Thanks.
b
@gentle-hamburger-31302 - would you happen to have any ideas here?
g
No, I am using Ubuntu
b
What is the recommended way to do dev with datahub? Because following the instructions here: https://datahubproject.io/docs/developers I always run into different issues on all my computers 😩
Ok I managed to get it to build on my linux box running this: 
sudo ./gradlew build -x :docs-website:generateJsonSchema
More progress on the M1 front got this error while running: 
sudo ./gradlew build :metadata-ingestion-modules:airflow-plugin:lint
which was the last test to fail it looks like the airflow-plugin is causing the issue
Copy code
> Task :metadata-ingestion-modules:airflow-plugin:lint FAILED
+ find venv/lib -path '*airflow/_vendor/connexion/spec.py' -exec sed -i.bak -e '169,169s/  # type: List\[str\]//g' '{}' ';'
+ source venv/bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/Users/benmarte/Documents/github/personal/datahub/metadata-ingestion-modules/airflow-plugin/venv
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH='/Users/benmarte/Documents/github/personal/datahub/smoke-test/venv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech <http://Preview.app/Contents/Public:/usr/local/go/bin:/Users/benmarte/.yarn/bin:/Users/benmarte/.config/yarn/global/node_modules/.bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/benmarte/.cargo/bin|Preview.app/Contents/Public:/usr/local/go/bin:/Users/benmarte/.yarn/bin:/Users/benmarte/.config/yarn/global/node_modules/.bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/benmarte/.cargo/bin>'
++ PATH='/Users/benmarte/Documents/github/personal/datahub/metadata-ingestion-modules/airflow-plugin/venv/bin:/Users/benmarte/Documents/github/personal/datahub/smoke-test/venv/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech <http://Preview.app/Contents/Public:/usr/local/go/bin:/Users/benmarte/.yarn/bin:/Users/benmarte/.config/yarn/global/node_modules/.bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/benmarte/.cargo/bin|Preview.app/Contents/Public:/usr/local/go/bin:/Users/benmarte/.yarn/bin:/Users/benmarte/.config/yarn/global/node_modules/.bin:/opt/homebrew/opt/openssl@3/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/benmarte/.cargo/bin>'
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' 'x(venv) ' '!=' x ']'
++ PS1='(venv) '
++ export PS1
++ '[' -n /bin/bash -o -n '' ']'
++ hash -r
+ black --check --diff src/ tests/
Traceback (most recent call last):
  File "/Users/benmarte/Documents/github/personal/datahub/metadata-ingestion-modules/airflow-plugin/venv/bin/black", line 5, in <module>
    from black import patched_main
ImportError: dlopen(/Users/benmarte/Documents/github/personal/datahub/metadata-ingestion-modules/airflow-plugin/venv/lib/python3.8/site-packages/black/__init__.cpython-38-darwin.so, 0x0002): tried: '/Users/benmarte/Documents/github/personal/datahub/metadata-ingestion-modules/airflow-plugin/venv/lib/python3.8/site-packages/black/__init__.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64)))
b
Thanks for this info. @gray-shoe-75895 @dazzling-judge-80093 any guidance on these airflow issues for M1?
d
hmm, it seems like black fails here. Can you try to reinstall black?
b
@dazzling-judge-80093 I can if you send me the command to install it 🙏
d
pip install --force-reinstall black
b
Cool I’ll try that in a bit thanks
Nope still getting this:
Copy code
Failed to build python-ldap cx-Oracle
ERROR: Could not build wheels for python-ldap, cx-Oracle, which is required to install pyproject.toml-based projects

> Task :metadata-ingestion:installDevTest FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':metadata-service:auth-ranger-impl:test'.
> There were failing tests. See the report at: file:///Users/benmarte/Documents/github/personal/datahub/metadata-service/auth-ranger-impl/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':metadata-ingestion:installDevTest'.
> Process 'command 'bash'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at <https://help.gradle.org>

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See <https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings>

BUILD FAILED in 5m 50s
247 actionable tasks: 239 executed, 8 up-to-date
f
Hey @busy-dusk-4970, try to add this prefix, or open a terminal using rosetta. Note: make sure you already had java 8 installed on your Mac M1. (https://www.azul.com/downloads/?version=java-8-lts&amp;os=macos&amp;architecture=arm-64-bit&amp;package=jdk)
Copy code
arch -x86_64 ./gradlew build
:metadata-service:auth-ranger-impl:test --stacktrace --debug
b
Awesome I'll try that and repot back thanks
d
well, rosetta never worked for me properly 😞 @busy-dusk-4970 what command do you try to run exactly? if it is the gradle build then try without running test:
Copy code
./gradlew build -x test -x yarnTest
b
@dazzling-judge-80093 I ran 
sudo arch -x86_64 ./gradlew build
for some reason if I don't use sudo it gives a different error where it tries to delete some directory using sudo avoids that problem
d
I think we should avoid 
arch -x86_64
as in my experience the rosetta doesn’t work very well
b
That worked @dazzling-judge-80093 I was able to build thanks
d
awesome 🙂
f
@dazzling-judge-80093 it’s failed in 
metadata-ingestion:docGen
Could you please take a look?
Copy code
08/23/2022 08:04:07 PM ERROR:Failed to find source classes for platform bigquery. Did you remember to annotate your source class with @platform_name(bigquery)?
Failed to process bigquery due to exception
Traceback (most recent call last):
ImportError("dlopen(/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): tried: '/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cygrpc.cpython-310-darwin.so' (no such file), '/usr/lib/cygrpc.cpython-310-darwin.so' (no such file)")
Failed to process bigquery-usage due to exception
  File "/Users/146685/Workspace/datahub/metadata-ingestion/scripts/docgen.py", line 773, in <module>
ImportError("dlopen(/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so, 0x0002): tried: '/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/grpc/_cython/cygrpc.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/cygrpc.cpython-310-darwin.so' (no such file), '/usr/lib/cygrpc.cpython-310-darwin.so' (no such file)")
    generate()
  File "/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/146685/Workspace/datahub/metadata-ingestion/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/146685/Workspace/datahub/metadata-ingestion/scripts/docgen.py", line 653, in generate
    f"There are {len(platform_docs['plugins'].keys())} sources that provide integration with {platform_docs['name']}\n"
KeyError: 'name'

> Task :metadata-ingestion:docGen FAILED
d
It seems like 
grpc
lib was installed for the wrong architecture. Can you try to remove and reinstall it?
f
I tried, but metadata-ingestion:docGen stills failed with grpc error. It seems like a bug in gprc :/
Copy code
grpcio 1.44.0
Python 3.10.6
g
Could you try running 
./gradlew :metadata:ingestion:clean
first and then trying again. It looks like part of the python venv environment was installed with arm instead of x86
f
Clean is Ok. But the error stills there. Here is the command which was failed.
Copy code
./gradlew build -x test -x yarnTest
b
We’ve similar issue with grpcio on the task 
:metadata-ingestion:docGen
when build on M1 with Python 3.10. Reinstalling grpcio from source distribution works.
Copy code
pip uninstall grpcio
pip install --no-binary :all: grpcio==1.44.0
Related issue - https://github.com/grpc/grpc/issues/28387#issuecomment-1003867850
f
Brilliant! Thanks @boundless-student-48844
3 Views
Open in Slack
PreviousNext
Powered by