Hi team, I have a problem. When I tried to connect to the hive data source using datahub, I was prompted that the connection could not be connected The port number for connecting to hive is 10000

Hey there! 👋 Make sure your message includes the following information if relevant, so we can help more effectively! 1. Which DataHub version are you using? (e.g. 0.12.0) 2. How are you deploying DataHub? (e.g. Helm, Quickstart, etc)

1. 0.10.4

Can you show more logs? Have you checked connect to Hive database?.

~~ Execution Summary - RUN_INGEST ~~ Execution finished with errors. {'exec_id': '3247f27e-6678-4430-b978-081933e43a9a', 'infos': ['2023-11-13 092939.580012 INFO: Starting execution for task with name=RUN_INGEST', "2023-11-13 093015.831415 INFO: Failed to execute 'datahub ingest'", '2023-11-13 093015.831595 INFO: Caught exception EXECUTING task_id=3247f27e-6678-4430-b978-081933e43a9a, name=RUN_INGEST, ' 'stacktrace=Traceback (most recent call last):\n' ' File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 122, in execute_task\n' ' task_event_loop.run_until_complete(task_future)\n' ' File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete\n' ' return future.result()\n' ' File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 231, in execute\n' ' raise TaskError("Failed to execute \'datahub ingest\'")\n' "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"], 'errors': []} ~~ Ingestion Report ~~ { "cli": { "cli_version": "0.10.4", "cli_entry_location": "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/__init__.py", "py_version": "3.10.10 (main, Mar 14 2023, 023711) [GCC 10.2.1 20210110]", "py_exec_path": "/tmp/datahub/ingest/venv-hive-0.10.4/bin/python3", "os_details": "Linux-4.19.91-26.al7.x86_64-x86_64-with-glibc2.31", "peak_memory_usage": "84.78 MB", "mem_info": "84.78 MB", "peak_disk_usage": "33.92 GB", "disk_info": { "total": "528.31 GB", "used": "33.92 GB", "free": "472.69 GB" } }, "source": { "type": "hive", "report": { "events_produced": 0, "events_produced_per_sec": 0, "entities": {}, "aspects": {}, "warnings": {}, "failures": {}, "soft_deleted_stale_entities": [], "tables_scanned": 0, "views_scanned": 0, "entities_profiled": 0, "filtered": [], "start_time": "2023-11-13 092941.805524 (32.66 seconds ago)", "running_time": "32.66 seconds" } }, "sink": { "type": "datahub-rest", "report": { "total_records_written": 0, "records_written_per_second": 0, "warnings": [], "failures": [], "start_time": "2023-11-13 092941.547752 (32.92 seconds ago)", "current_time": "2023-11-13 093014.465522 (now)", "total_duration_in_seconds": 32.92, "gms_version": "v0.10.4", "pending_requests": 0 } } } ~~ Ingestion Logs ~~ Obtaining venv creation lock... Acquired venv creation lock venv setup time = 0 This version of datahub supports report-to functionality datahub --debug ingest run -c /tmp/datahub/ingest/3247f27e-6678-4430-b978-081933e43a9a/recipe.yml --report-to /tmp/datahub/ingest/3247f27e-6678-4430-b978-081933e43a9a/ingestion_report.json [2023-11-13 092940,454] DEBUG {datahub.telemetry.telemetry:219} - Sending init Telemetry [2023-11-13 092941,184] DEBUG {datahub.telemetry.telemetry:248} - Sending telemetry for function-call [2023-11-13 092941,514] INFO {datahub.cli.ingest_cli:173} - DataHub CLI version: 0.10.4 [2023-11-13 092941,555] DEBUG {datahub.ingestion.sink.datahub_rest:118} - Setting env variables to override config [2023-11-13 092941,555] DEBUG {datahub.ingestion.sink.datahub_rest:120} - Setting gms config [2023-11-13 092941,556] DEBUG {datahub.ingestion.run.pipeline:209} - Sink type datahub-rest (<class 'datahub.ingestion.sink.datahub_rest.DatahubRestSink'>) configured [2023-11-13 092941,556] INFO {datahub.ingestion.run.pipeline:210} - Sink configured successfully. DataHubRestEmitter: configured to talk to http://datahub-datahub-gms:8080 [2023-11-13 092941,566] DEBUG {datahub.ingestion.run.pipeline:284} - Reporter type:file,<class 'datahub.ingestion.reporting.file_reporter.FileReporter'> configured. [2023-11-13 092941,805] DEBUG {datahub.ingestion.source.state.stateful_ingestion_base:252} - Successfully created datahub state provider. [2023-11-13 092941,806] DEBUG {datahub.telemetry.telemetry:248} - Sending telemetry for sql_config [2023-11-13 092942,103] DEBUG {datahub.telemetry.telemetry:248} - Sending telemetry for sql_profiling_config [2023-11-13 092942,411] DEBUG {datahub.ingestion.run.pipeline:226} - Source type hive (<class 'datahub.ingestion.source.sql.hive.HiveSource'>) configured [2023-11-13 092942,411] INFO {datahub.ingestion.run.pipeline:227} - Source configured successfully. [2023-11-13 092942,413] INFO {datahub.cli.ingest_cli:129} - Starting metadata ingestion [2023-11-13 092942,413] DEBUG {datahub.ingestion.source.sql.sql_common:97} - sql_alchemy_url=hive://hadoop@xxx.xxx.xxx.xxx:10000/tem [2023-11-13 093014,464] ERROR {thrift.transport.TSocket:145} - Could not connect to any of [('xxx.xxx.xxx.xxx', 10000)] [2023-11-13 093014,465] INFO {datahub.ingestion.reporting.file_reporter:52} - Wrote UNKNOWN report successfully to <_io.TextIOWrapper name='/tmp/datahub/ingest/3247f27e-6678-4430-b978-081933e43a9a/ingestion_report.json' mode='w' encoding='UTF-8'> [2023-11-13 093014,466] INFO {datahub.cli.ingest_cli:134} - Source (hive) report: {'events_produced': 0, 'events_produced_per_sec': 0, 'entities': {}, 'aspects': {}, 'warnings': {}, 'failures': {}, 'soft_deleted_stale_entities': [], 'tables_scanned': 0, 'views_scanned': 0, 'entities_profiled': 0, 'filtered': [], 'start_time': '2023-11-13 092941.805524 (32.66 seconds ago)', 'running_time': '32.66 seconds'} [2023-11-13 093014,466] INFO {datahub.cli.ingest_cli:137} - Sink (datahub-rest) report: {'total_records_written': 0, 'records_written_per_second': 0, 'warnings': [], 'failures': [], 'start_time': '2023-11-13 092941.547752 (32.92 seconds ago)', 'current_time': '2023-11-13 093014.466378 (now)', 'total_duration_in_seconds': 32.92, 'gms_version': 'v0.10.4', 'pending_requests': 0} [2023-11-13 093014,467] DEBUG {datahub.telemetry.telemetry:248} - Sending telemetry for function-call [2023-11-13 093014,796] ERROR {datahub.entrypoints:199} - Command failed: Could not connect to any of [('xxx.xxx.xxx.xxx', 10000)] Traceback (most recent call last): File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/entrypoints.py", line 186, in main sys.exit(datahub(standalone_mode=False, **kwargs)) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(*args, **kwargs) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 379, in wrapper raise e File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/telemetry/telemetry.py", line 334, in wrapper res = func(*args, **kwargs) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/utilities/memory_leak_detector.py", line 95, in wrapper return func(ctx, *args, **kwargs) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 198, in run loop.run_until_complete(run_func_check_upgrade(pipeline)) File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 158, in run_func_check_upgrade ret = await the_one_future File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 149, in run_pipeline_async return await loop.run_in_executor( File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 140, in run_pipeline_to_completion raise e File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/cli/ingest_cli.py", line 132, in run_pipeline_to_completion pipeline.run() File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/run/pipeline.py", line 364, in run for wu in itertools.islice( File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 114, in auto_stale_entity_removal for wu in stream: File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 138, in auto_workunit_reporter for wu in stream: File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 202, in auto_browse_path_v2 for urn, batch in _batch_workunits_by_urn(stream): File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 282, in _batch_workunits_by_urn for wu in stream: File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 151, in auto_materialize_referenced_tags for wu in stream: File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/api/source_helpers.py", line 65, in auto_status_aspect for wu in stream: File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/source/sql/sql_common.py", line 482, in get_workunits_internal for inspector in self.get_inspectors(): File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/datahub/ingestion/source/sql/two_tier_sql_source.py", line 99, in get_inspectors with engine.connect() as conn: File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3325, in connect return self._connection_cls(self, close_with_result=close_with_result) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 96, in init else engine.raw_connection() File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection return self._wrap_pool_connect(self.pool.connect, _connection) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect return fn() File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 327, in connect return _ConnectionFairy._checkout(self) File "/tmp/datahub/ingest/venv-hive-0.10.4/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout fairy = _ConnectionRecord.checkout(pool)

Hi @straight-apartment-51931, Have you checked connect to Hive database?. I guess that is a network issue.

Yes, I checked it

I haven't tried with Hive yet. But, I guess that got an issue about network with port 10000 based on your log.

ok i will try again. thank you

How do you deploy Datahub? By helm on K8S?

K8S

I usually to use a

busy-box

pod to check the connection with

telnet

before. If everything is good, I think we need support from others.

Thank you. After whitelisting port 10000, can successfully connect to hive

It's sound great ! 😆

I have another problem. When I use datahub to connect to hive, multiple connections occur, occupying queue resources of yarn. Have you ever encountered this problem?