Hello
Can we specify a hive queue name while ing...
# ingestionr
rich-policeman-92383
12/27/2021, 12:32 PMHello
Can we specify a hive queue name while ingestion metadata from hive source.
Like in beeline we can do something like:
Copy code
beeline -e "set tez.queue.name='myqueue' describe formatted myschem.mytable;"d
dazzling-judge-80093
12/27/2021, 12:37 PMI’m not sure if this works but can you try to specify this at the connections args in your config?
Copy code
options:
connect_args:
'tez.queue.name': '<queue-name>'r
rich-policeman-92383
12/27/2021, 12:38 PMokay thanks i will try it and share the outcome.
d
dazzling-judge-80093
12/27/2021, 12:38 PMthanks
r
rich-policeman-92383
12/28/2021, 11:28 AMTried the suggestion but it failed with below error
Copy code
connection = pool._invoke_creator(self)
File "/dhubv08_19/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/dhubv08_19/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/IngestionRecipies/dhubv08_19/lib64/python3.6/site-packages/pyhive/hive.py", line 126, in connect
return Connection(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'tez.queue.name'
Run ended on Tue Dec 28 16:52:32 IST 202rich-policeman-92383
12/28/2021, 11:29 AMConfig Details:
Copy code
- ".*dump.*"
- ".*test.*"
options:
connect_args: {'auth': 'KERBEROS','kerberos_service_name': 'hive', 'tez.queue.name': 'hpurge'}d
dazzling-judge-80093
12/28/2021, 12:33 PMOuch, I will check how we could squeeze in this parameter
dazzling-judge-80093
12/28/2021, 4:51 PMPlease, can you try this way?
Copy code
options:
connect_args:
'auth': 'KERBEROS'
'kerberos_service_name': 'hive'
configuration:
'tez.queue.name': 'hpurge'dazzling-judge-80093
12/28/2021, 4:52 PMthis config should pass but I couldn’t test is as I don’t have actual hive setup with tez
r
rich-policeman-92383
12/31/2021, 11:33 AMYes sure i will try the config shared by you and share the outcome.
👍 1