https://datahubproject.io logo
Join Slack
Powered by
Hi there, I want to send hive DB to datahub, but I...
# ingestion
s
Hi there, I want to send hive DB to datahub, but I keep getting errors. The connection URL used by datagrip is 
jdbc:<hive2://hive_host:10001/;transportMode=http;httpPath=cliservice>
It's a recipe file
Copy code
source:
    type: hive
    config:
      host_port: hive_host:10001
      database: db_name
      username: id
      password: pw
      options:
       connect_args:
          http_path: "/cliservice"
          auth: LDAP

sink:
    type: datahub-rest
    config:
      server: "<http://localhost:8080>"
error message is
Copy code
[2022-07-12 12:09:54,931] ERROR    {datahub.entrypoints:184} - File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/entrypoints.py", line 149, in main

....

'---- (full traceback above) ----
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/entrypoints.py", line 149, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/upgrade/upgrade.py", line 333, in wrapper
    res = func(*args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/telemetry/telemetry.py", line 338, in wrapper
    raise e
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/telemetry/telemetry.py", line 290, in wrapper
    res = func(*args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/utilities/memory_leak_detector.py", line 102, in wrapper
    res = func(*args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/cli/ingest_cli.py", line 131, in run
    raise e
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/cli/ingest_cli.py", line 117, in run
    pipeline.run()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/ingestion/run/pipeline.py", line 217, in run
    self.preview_workunits if self.preview_mode else None,
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/ingestion/source/sql/sql_common.py", line 712, in get_workunits
    for inspector in self.get_inspectors():
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/ingestion/source/sql/sql_common.py", line 516, in get_inspectors
    with engine.connect() as conn:
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2263, in connect
    return self._connection_cls(self, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 104, in __init__
    else engine.raw_connection()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2370, in raw_connection
    self.pool.unique_connection, _connection
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
    return fn()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
    return _ConnectionFairy._checkout(self)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    with_traceback=exc_tb,
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
    return self.dbapi.connect(*cargs, **cparams)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/pyhive/hive.py", line 126, in connect
    return Connection(*args, **kwargs)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/pyhive/hive.py", line 267, in __init__
    self._transport.open()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/thrift_sasl/__init__.py", line 93, in open
    status, payload = self._recv_sasl_message()
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/thrift_sasl/__init__.py", line 115, in _recv_sasl_message
    payload = self._trans_read_all(length)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/thrift_sasl/__init__.py", line 210, in _trans_read_all
    return read_all(sz)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/thrift/transport/TTransport.py", line 62, in readAll
    chunk = self.read(sz - have)
File "/users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/thrift/transport/TSocket.py", line 167, in read
    message='TSocket read 0 bytes')

TTransportException: TSocket read 0 bytes
[2022-07-12 12:09:54,942] INFO     {datahub.entrypoints:188} - DataHub CLI version: 0.8.40.2 at /users/user/workspace/datahub/datahub-env/lib64/python3.6/site-packages/datahub/__init__.py
[2022-07-12 12:09:54,942] INFO     {datahub.entrypoints:191} - Python version: 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] at /users/user/workspace/datahub/datahub-env/bin/python3 on Linux-3.10.0-693.2.2.el7.x86_64-x86_64-with-centos-7.9.2009-Core
[2022-07-12 12:09:54,942] INFO     {datahub.entrypoints:193} - GMS config {'models': {}, 'versions': {'linkedin/datahub': {'version': 'v0.8.40', 'commit': '5bb7fe3691e153ff64137a8bdd64ec1473b6095f'}}, 'managedIngestion': {'defaultCliVersion': '0.8.40', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'datasetUrnNameCasing': False, 'retention': 'true', 'datahub': {'serverType': 'quickstart'}, 'noCode': 'true'}
Please, help me...
c
@steep-vr-39297 Please try below things: 1. 
http_path: "cliservice"
2. _`scheme: 'hive+http'`_scheme should be set as well in config, as mentioned in source doc to set proper transport mode
s
@careful-pilot-86309 
http_path: "cliservice"
is`Port could not be cast to integer value as '20001cliservice'`
Copy code
options:
    connect_args:
        auth: BASIC
scheme: 'hive+http'
There is an error, so when I change it to BASIC and run it, a 
CannotSendHeader
error occurs.
Copy code
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/pyhive/hive.py", line 273, in __init__
    response = self._client.OpenSession(open_session_req)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/TCLIService/TCLIService.py", line 186, in OpenSession
    self.send_OpenSession(req)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/TCLIService/TCLIService.py", line 195, in send_OpenSession
    self._oprot.trans.flush()
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/pyhive/hive.py", line 81, in flush
    super(TCookieHttpClient, self).flush()
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/thrift/transport/THttpClient.py", line 191, in flush
    self.__http.putheader('Cookie', self.headers['Set-Cookie'])
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/http/client.py", line 1218, in putheader
    raise CannotSendHeader()

CannotSendHeader
Copy code
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/thrift/transport/TTransport.py", line 62, in readAll
    58   def readAll(self, sz):
    59       buff = b''
    60       have = 0
    61       while (have < sz):
--> 62           chunk = self.read(sz - have)
    63           chunkLen = len(chunk)
    ..................................................
     self = <thrift.transport.TSocket.TSocket object at 0x7f85d914fcd0>
     sz = 1414811695
     buff = b'1.1 400 Illegal character 0x1\r\nContent-Length: 0\r\nConnection: close\r\nServer: Jetty(9.3.25.v20180904)\r\n\r\n'
     have = 104
     chunk = b'1.1 400 Illegal character 0x1\r\nContent-Length: 0\r\nConnection: close\r\nServer: Jetty(9.3.25.v20180904)\r\n\r\n'
     self.read = <method 'TSocket.read' of <thrift.transport.TSocket.TSocket object at 0x7f85d914fcd0> TSocket.py:148>
     chunkLen = 104
    ..................................................

File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/thrift/transport/TSocket.py", line 166, in read
    148  def read(self, sz):
 (...)
    162              raise TTransportException(type=TTransportException.TIMED_OUT, message="read timeout", inner=e)
    163          else:
    164              raise TTransportException(message="unexpected exception", inner=e)
    165      if len(buff) == 0:
--> 166          raise TTransportException(type=TTransportException.END_OF_FILE,
    167                                    message='TSocket read 0 bytes')
    ..................................................
     self = <thrift.transport.TSocket.TSocket object at 0x7f85d914fcd0>
     sz = 1414811591
     TTransportException = <class 'thrift.transport.TTransport.TTransportException'>
     TTransportException.TIMED_OUT = 3
     buff = b''
     TTransportException.END_OF_FILE = 4
    ..................................................
"sz" is 1414811695, but "buff" is 1.1400 Illegal character.
h
I think, from your original recipe shared earlier, you can remove http_path. it seems to be already set , default should work - https://github.com/acryldata/PyHive/blob/master/pyhive/hive.py#L145 still figuring out how to correctly set transport mode. Probably its taking binary by default.
s
@hundreds-photographer-13496 Thank you for your answer. The hiveserver2 I use only supports 
trnasportMode=http
Does the sqlalchemy hive package not support 
transportMode=http
? As Hardikar told me, I added 
scheme: hive+http
, but an error occurs. recipe.yml:
Copy code
source:
    type: hive
    config:
      scheme: 'hive+http'
      host_port: host:10000
      database: default
      username: id
      password: pw
      options:
       connect_args:
          auth: NONE

sink:
    type: datahub-rest
    config:
      server: "<http://localhost:8080>"
error message :
Copy code
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect
    return self.dbapi.connect(*cargs, **cparams)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/pyhive/hive.py", line 126, in connect
    return Connection(*args, **kwargs)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/pyhive/hive.py", line 273, in __init__
    response = self._client.OpenSession(open_session_req)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/TCLIService/TCLIService.py", line 186, in OpenSession
    self.send_OpenSession(req)
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/TCLIService/TCLIService.py", line 195, in send_OpenSession
    self._oprot.trans.flush()
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/pyhive/hive.py", line 81, in flush
    super(TCookieHttpClient, self).flush()
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/site-packages/thrift/transport/THttpClient.py", line 191, in flush
    self.__http.putheader('Cookie', self.headers['Set-Cookie'])
File "/Users/sseoju/miniconda3/envs/datahub/lib/python3.8/http/client.py", line 1218, in putheader
    raise CannotSendHeader()

CannotSendHeader
h
Hi @steep-vr-39297 It seems that setting 
scheme:hive_http
along with 
auth:LDAP
is not supported by hive sqlalchemy dialect. can you confirm what type of authentication is supported by your hive server ? Are you certain that it's LDAP ?
s
@hundreds-photographer-13496 Yes, It is set to 
LDAP
in 
hive-site.xml
If 
scheme:hive_http
an error occurs. -> 
NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:hive_http
Do I need other settings for 
have-site.xml
?
h
oops I didn't mean 
scheme:hive_http
I meant 
scheme:hive+http
Are you able able to ingest if you use both http scheme and auth LDAP ?
Copy code
source:
    type: hive
    config:
      scheme: hive+http
      host_port: hive_host:10001
      database: db_name
      username: id
      password: pw
      options:
       connect_args:
          http_path: "/cliservice"
          auth: LDAP

sink:
    type: datahub-rest
    config:
      server: "<http://localhost:8080>"
s
error message when executing the above command => 
Authentication is not valid use one of:BASIC, NOSASL, KERBEROS, NONE
h
right, exactly what I expected. Can you confirm the transport mode enabled on your hive server. ( 
hive.server2.transport.mode
property )
s
hive.server2.transport.mode
is http
h
ooh. So the thing is I can't seem to find a way to set http transport mode when auth is LDAP through sqlalchemy hive. That may need some changes in sqlalchemy hive. My guess is that the socket error is because sqlalchemy client is trying to connect over binary transport mode whereas server only responds http transport mode. Is it possible to enable binary transport mode on hive server and retry ingestion ? I believe setting 
hive.server2.transport.mode
to 
all
should support both http and binary transport modes (ref: https://issues.apache.org/jira/browse/HIVE-5312) You may need to restart hive server. Setting 
hive.server2.transport.mode
to `binary`is also an option, however it may break other clients to hive server.
s
Okey, I'll give it a try and tell you.
Wow! I succeeded! Thanks a lot! 👍😆
h
Awesome ! thanks for the update !!
Open in Slack
PreviousNext
Powered by