https://datahubproject.io logo
Join Slack
Powered by
I use a docker deployed <datahub.My> component ver...
# all-things-deployment
b
I use a docker deployed datahub.My component version as shown below : Docker version 1.13.1 acryl-datahub, version 0.8.33.3 hadoop3.1.4 hive3.1.0 My recipes is shown below ： 
source:
type: hive config: host_port: "192.168.127.137:2181" username: 'root' password: '123456' sink: type: "datahub-rest" config: server: "http://192.168.88.129:8080" My way of execution : 
python -m  datahub ingest -c hive_to_datahub_rest.yml
But the error. Who can help diagnose the problem appears. Thank you for your attention!
Copy code
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    648  def __connect(self, first_connect_check=False):
 (...)
    652      # creator fails, this attribute stays None
    653      self.connection = None
    654      try:
    655          self.starttime = time.time()
--> 656          connection = pool._invoke_creator(self)
    657          pool.logger.debug("Created new connection %r", connection)
    ..................................................
     self = <sqlalchemy.pool.base._ConnectionRecord object at 0x7f20f54ad2b0>
     first_connect_check = True
     self.connection = None
     self.starttime = 1651741379.4730806
     pool._invoke_creator = <function 'DefaultEngineStrategy.create.<locals>.connect' strategies.py:106>
     pool.logger.debug = <method 'Logger.debug' of <Logger sqlalchemy.pool.impl.QueuePool (WARNING)> __init__.py:1412>
    ..................................................

File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    106  def connect(connection_record=None):
 (...)
    110                  dialect, connection_record, cargs, cparams
    111              )
    112              if connection is not None:
    113                  return connection
--> 114      return dialect.connect(*cargs, **cparams)
    ..................................................
     connection_record = <sqlalchemy.pool.base._ConnectionRecord object at 0x7f20f54ad2b0>
     dialect = <pyhive.sqlalchemy_hive.HiveDialect object at 0x7f20f54e5850>
     cargs = []
     cparams = {'host': '192.168.127.137',
                'port': 2181,
                'username': 'root',
                'password': '123456',
                'database': 'default'}
     dialect.connect = <method 'DefaultDialect.connect' of <pyhive.sqlalchemy_hive.HiveDialect object at 0x7f20f54e5850> default.py:506>
    ..................................................

File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect
    506  def connect(self, *cargs, **cparams):
    507      # inherits the docstring from interfaces.Dialect.connect
--> 508      return self.dbapi.connect(*cargs, **cparams)
    ..................................................
     self = <pyhive.sqlalchemy_hive.HiveDialect object at 0x7f20f54e5850>
     cargs = ()
     cparams = {'host': '192.168.127.137',
                'port': 2181,
                'username': 'root',
                'password': '123456',
                'database': 'default'}
     self.dbapi.connect = <function 'connect' hive.py:120>
    ..................................................

File "/usr/local/python3/lib/python3.8/site-packages/pyhive/hive.py", line 126, in connect
    120  def connect(*args, **kwargs):
 (...)
    122      arguments.
    123
    124      :returns: a :py:class:`Connection` object.
    125      """
--> 126      return Connection(*args, **kwargs)
    ..................................................
     args = ()
     kwargs = {'host': '192.168.127.137',
               'port': 2181,
               'username': 'root',
               'password': '123456',
               'database': 'default'}
     Connection = <class 'pyhive.hive.Connection'>
    ..................................................

File "/usr/local/python3/lib/python3.8/site-packages/pyhive/hive.py", line 198, in __init__
    132  def __init__(
    133      self,
    134      host=None,
    135      port=None,
    136      scheme=None,
    137      username=None,
    138      database='default',
    139      auth=None,
    140      configuration=None,
    141      kerberos_service_name=None,
    142      password=None,
    143      check_hostname=None,
    144      ssl_cert=None,
    145      http_path='/cliservice/',
    146      thrift_transport=None
    147  ):
 (...)
    194      username = username or getpass.getuser()
    195      configuration = configuration or {}
    196
    197      if (password is not None) != (auth in ('LDAP', 'CUSTOM')):
--> 198          raise ValueError("Password should be set if and only if in LDAP or CUSTOM mode; "
    199                           "Remove password or use one of those modes")
    ..................................................
     self = <pyhive.hive.Connection object at 0x7f20f54ad790>
     host = '192.168.127.137'
     port = 2181
     scheme = None
     username = 'root'
     database = 'default'
     auth = None
     configuration = {}
     kerberos_service_name = None
     password = '123456'
     check_hostname = None
     ssl_cert = None
     http_path = '/cliservice/'
     thrift_transport = None
     getpass.getuser = <function 'getuser' getpass.py:154>
    ..................................................

---- (full traceback above) ----
File "/usr/local/python3/lib/python3.8/site-packages/datahub/entrypoints.py", line 149, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/usr/local/python3/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
File "/usr/local/python3/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/python3/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/python3/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/python3/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 317, in wrapper
    raise e
File "/usr/local/python3/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 269, in wrapper
    res = func(*args, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/datahub/utilities/memory_leak_detector.py", line 102, in wrapper
    res = func(*args, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 128, in run
    raise e
File "/usr/local/python3/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 114, in run
    pipeline.run()
File "/usr/local/python3/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 214, in run
    for wu in itertools.islice(
File "/usr/local/python3/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 677, in get_workunits
    for inspector in self.get_inspectors():
File "/usr/local/python3/lib/python3.8/site-packages/datahub/ingestion/source/sql/sql_common.py", line 481, in get_inspectors
    with engine.connect() as conn:
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2263, in connect
    return self._connection_cls(self, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 104, in __init__
    else engine.raw_connection()
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2369, in raw_connection
    return self._wrap_pool_connect(
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
    return fn()
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
    return _ConnectionFairy._checkout(self)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
File "/usr/local/python3/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect
    return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/python3/lib/python3.8/site-packages/pyhive/hive.py", line 126, in connect
    return Connection(*args, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/pyhive/hive.py", line 198, in __init__
    raise ValueError("Password should be set if and only if in LDAP or CUSTOM mode; "

ValueError: Password should be set if and only if in LDAP or CUSTOM mode; Remove password or use one of those modes
[2022-05-05 05:03:00,472] INFO     {datahub.entrypoints:176} - DataHub CLI version: 0.8.33.3 at /usr/local/python3/lib/python3.8/site-packages/datahub/__init__.py
[2022-05-05 05:03:00,473] INFO     {datahub.entrypoints:179} - Python version: 3.8.3 (default, May  2 2022, 02:44:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] at /usr/bin/python on Linux-3.10.0-1160.62.1.el7.x86_64-x86_64-with-glibc2.17
[2022-05-05 05:03:00,473] INFO     {datahub.entrypoints:182} - GMS config {'models': {}, 'versions': {'linkedin/datahub': {'version': 'v0.8.33', 'commit': '2b62ed5260157987d5053ebc795b10c43177fad5'}}, 'managedIngestion': {'defaultCliVersion': '0.8.32.1', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'datasetUrnNameCasing': False, 'retention': 'true', 'noCode': 'true'}
d
I think you should set auth mechanism LDAP if you want to use username/password authentication
Copy code
config:
        host_port: "192.168.127.137:2181"
        username: 'root'
        password: '123456'
       options:
         connect_args:
           auth: LDAP
b
thank you very much. Your suggestion reminded me. I should not use account password authentication. I changed the configuration. And updated the content of the, and my problem has been solved. 
yum install cyrus-sasl-plain  cyrus-sasl-devel  cyrus-sasl-gssapi
2 Views
Open in Slack
PreviousNext
Powered by