https://datahubproject.io logo
Join Slack
Powered by
Hi everyone, I am facing a weird (and annoying) i...
# troubleshoot
b
Hi everyone, I am facing a weird (and annoying) issue, which I'm not sure whether its to do with Datahub or Airflow (MWAA). I am trying to setup the airflow integration using kafka connection id. I followed all the steps on airflow integration documentation, but made a the mistake of setting connection id as 
datahub_rest_default
while using 
datahub_kafka
connection type. Obviously this caused issues while emitting metadata towards the end of my job. I noticed this and corrected the connection id to 
datahub_kafka_default
. However, either airflow or the datahub plugin is not picking up on this change and is continuously looking for 
datahub_rest_default
connection id instead of the corrected one. I deleted the connection and set it up again from scratch, but no luck.
Copy code
[2023-02-17, 08:12:03 UTC] {{logging_mixin.py:109}} INFO - Exception: Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/datahub_provider/_plugin.py", line 284, in custom_on_success_callback
    datahub_task_status_callback(context, status=InstanceRunResult.SUCCESS)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/datahub_provider/_plugin.py", line 136, in datahub_task_status_callback
    DatahubGenericHook(context["_datahub_config"].datahub_conn_id)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/datahub_provider/hooks/datahub.py", line 189, in get_underlying_hook
    conn = self.get_connection(self.datahub_conn_id)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/hooks/base.py", line 68, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 410, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `datahub_rest_default` isn't defined
Any help with this much appreciated. PS: I tested this whole setup on a local instance of airflow first before I set it up on AWS MWAA. I did not face this issue on my local setup. It picked up kafka default connection and emitted metadata. (But I did not make the mistakes I did while configuring the connection id in my local)
đź‘€ 1
Screenshot 2023-02-17 at 4.27.29 PM.png
Screenshot 2023-02-17 at 4.27.50 PM.png
d
how is your airflow config’s datahub config part looks like?
b
I dont have one. When I tested it on my laptop, I did not have to specify anything in my airflow.cfg.
Are you referring to the value I have for 
core.lazy_load_plugins
- ? If yes, then its set to false.
Screenshot 2023-02-17 at 5.01.52 PM.png
I did not recall taking step 4 in my local setup and I have not done this on AWS MWAA as well. Is this the datahub airflow config you're referring to @dazzling-judge-80093?
d
In the 
Airflow configuration options
can you add this on the mwaa ui?
Copy code
datahub.conn_id: datahub_kafka_default
b
Sure. Let me do that.
Its gonna take almost 30 mins for airflow to update. I'll get back to you here once its applied!
d
sure, yeah, it takes some time
b
This is the error I see now.
airflow_mwaa_kafka_error
This looks like the same error I first saw when I said I made the mistake of accidentally setting the connection id as 
datahub_rest_default
. No, it doesnt.
To share some more context about my setup (for the purposes of POC) - I have used the quickstart guide to setup datahub on an EC2 instance. I have opened the necessary ports needed to access the UI as well as kafka in this case.
I've already successfully ingested metadata from from my warehouse and BI tools. Now i'm looking to setup airflow ingestion.
Hi @dazzling-judge-80093.. Any thoughts on the error above? Today I also noticed that after installing 
acryl-datahub-airflow-plugin
in airflow (MWAA), on 17th Feb, something went wrong with aiflow scheduler and many of our jobs did not run. AWS Support is looking into it to see what errors / what caused it. Reverting this change / uninstalling this plugin fixed our issues. FYI. Not sure if anyone else has also reported this issue.
To add more context about my setup here. I wanted to enable creation of Personal Access Token to send airflow metadata to datahub, so I followed this guide and set the following environment variable in the frontend and gms services. But I am not sure if I did it the right way. I had set this environment variable in the 
docker-compose-without-neo4j-quickstart.yml
file and used 
datahub docker quickstart --quickstart-compose-file <path to compose file>
command to start the service.
9 Views
Open in Slack
PreviousNext
Powered by