Namely, LDAP or Kerberos?
# ingestioni
incalculable-ocean-74010
03/01/2021, 5:34 PMNamely, LDAP or Kerberos?
b
big-carpet-38439
03/01/2021, 6:21 PMLDAP should be username / password right? I do not believe kerberos is supported yet, though it is on the roadmap @gray-shoe-75895 to confirm
g
gray-shoe-75895
03/01/2021, 6:49 PMI think Pedro is asking about using LDAP for authentication, not about ingesting from LDAP - we haven't tested them ourselves, but we use SQLAlchemy internally which should have support for it
👍 1
i
incalculable-ocean-74010
03/01/2021, 7:09 PMThat’s correct @gray-shoe-75895
incalculable-ocean-74010
03/01/2021, 7:10 PMAbout Kerberos I believe pyhive does not support everything. A colleague of mine opened a PR but it has not yet been merged: https://github.com/dropbox/PyHive/pull/325
incalculable-ocean-74010
03/01/2021, 7:12 PMI am under the impression that the ingestion framework may not work very with Kerberos or LDAP logins for hive since the connection string has a fixed format: https://github.com/linkedin/datahub/blob/cda1ce458974dda1cdc59b2c0957369e9524ea5e/metadata-ingestion/src/datahub/ingestion/source/sql_common.py#L62
incalculable-ocean-74010
03/01/2021, 7:12 PMWhat are your thoughts @gray-shoe-75895 ?
g
gray-shoe-75895
03/01/2021, 7:13 PMYep I think you're right - need to introduce more flexibility into that config model, or alternatively just accept an SQLAlchemy connection string
gray-shoe-75895
03/01/2021, 7:15 PMThe main fixes required are that the password should be optional and the connect arg options need to be more visible - did I understand that correctly?
i
incalculable-ocean-74010
03/01/2021, 7:41 PMSomething like that yes
incalculable-ocean-74010
03/01/2021, 7:43 PMI will try some hard coded changes (🔨) on my side to confirm if it works and let you know
✅ 1
incalculable-ocean-74010
03/02/2021, 5:12 PMHey @gray-shoe-75895 sorry for taking so long to come back to this. I was able to connect with kerberos after making the config model more flexible: here is the changes if you are interested: https://github.com/pedro93/datahub/commit/0e556e8eb7fba3ca6fc698a37d408d973fe72ead
incalculable-ocean-74010
03/02/2021, 5:12 PMIf you agree I can open a PR with these changes.
g
gray-shoe-75895
03/02/2021, 5:43 PMYep looks pretty good - opening a PR would be great. A minor nit is that we should use Optional[str] = None instead of the empty string
i
incalculable-ocean-74010
03/02/2021, 5:51 PMb
big-carpet-38439
03/02/2021, 7:55 PMthis is awesome, thanks pedro!
✅ 1