Hi guys, Need help with Airflow integration & Looker. Followed this documentation https://datahubproject.io/docs/lineage/airflow/ but when the dag runs, we are getting below error. Documentation did not talk about any authentication.

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/datahub/emitter/rest_emitter.py", line 241, in _emit_generic
    response.raise_for_status()
  File "/home/airflow/.local/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: <https://xxxx.company.com:8080/entities?action=ingest>

Similarly for looker, we are getting below error. Any thoughts?

datahub ingest run -c /tmp/datahub/ingest/18ded50c-08cb-4133-af8c-8b7c669b459e/recipe.yml
[2022-08-17 22:46:27,610] INFO     {datahub.cli.ingest_cli:99} - DataHub CLI version: 0.8.41
[2022-08-17 22:46:27,638] INFO     {datahub.ingestion.run.pipeline:160} - Sink configured successfully. DataHubRestEmitter: configured to talk to <http://xxx-xxx-datahub-datahub-gms:8080>
[2022-08-17 22:50:28,272] ERROR    {datahub.ingestion.run.pipeline:126} - Failed to initialize Looker client. Please check your configuration.
[2022-08-17 22:50:28,273] INFO     {datahub.cli.ingest_cli:115} - Starting metadata ingestion
[2022-08-17 22:50:28,273] INFO     {datahub.cli.ingest_cli:133} - Finished metadata pipeline

Failed to configure source (looker) due to Failed to initialize Looker client. Please check your configuration.
❗Client-Server Incompatible❗ Your client version 0.8.41 is older than your server version 0.8.42. Upgrading the cli to 0.8.42 is recommended.
 ➡️  Upgrade via "pip install 'acryl-datahub==0.8.42'"

Seem like you need to set up a token to use for authentication. When you set up the airflow connection config, you can put the token in the password field

That is right. I tried and it worked. Thank you 🙂

Awesome - I’ve also added some info about it to the docs https://github.com/datahub-project/datahub/pull/5703 🙂