Hi Is there some documented way to do local development for DataHub #contribute-code

Hi. Is there some documented way to do local devel...

average-vr-23088

02/27/2024, 8:56 PM

Hi. Is there some documented way to do local development for the Datahub Airflow plugin? I’m trying to spin up a local airflow via docker and mount the files for the airflow plugin into the airflow plugins directory. I’m generating the files after doing a

python setup.py build

on the datahub-airflow-plugin. Is this the right approach? I’ve also tried to install the necessary python dependencies but i’m getting errors from Airflow when it tries to load the plugin. I’m currently getting the following error:

Copy code

File "/usr/local/airflow/plugins/datahub_airflow_plugin/datahub_listener.py", line 19, in <module>
aws-mwaa-local-runner-2_7-local-runner-1  |     from datahub.sql_parsing.sqlglot_lineage import SqlParsingResult
aws-mwaa-local-runner-2_7-local-runner-1  | ModuleNotFoundError: No module named 'datahub.sql_parsing'

This is despite me installing the

acryl-datahub[sql-parser]==0.12.1.5

package. Any guidance would be much appreciated!

gray-shoe-75895

02/27/2024, 9:30 PM

You'll also need to build a wheel from source for the acryl-datahub package (in the metadata-ingestion directory)

average-vr-23088

02/27/2024, 9:30 PM

ahh. i see

average-vr-23088

02/27/2024, 9:30 PM

i’ll try that out. thanks!

average-vr-23088

02/27/2024, 11:08 PM

Awesome, i got it working with the wheel. Thanks! In case someone comes across this. Here is what i’m doing to test /iterate on the plugin with amazon’s managed Airflow. 1. Use the local mwaa runner 2. In the datahub repo, build the wheel for the ingestion module: a.

gradlew :metadata-ingestion:buildWheel

3. Use the mwaa startup script to install the wheel 4. Build the airflow plugin via

python setup.py build

and copy the built plugin folder to the mwaa

plugins

directory. 5. Update the requirements.txt to include:

Copy code

openlineage-airflow==1.2.0
gql
acryl-sqlglot==21.1.2.dev10

average-vr-23088

02/29/2024, 5:42 PM

@gray-shoe-75895 when you have some time i had a couple questions: 1. In the datahub airflow plugin, whats the relation between the DatahubLineageBackend and the DatahubListener? As far as i can tell, these two components emit MCEs completely independent of eachother. For newer airflow versions, do we only make use of the listener? 2. Is there some sort of logging config change required if you want to see logs for the listener hooks related to the task life cycle. For example, any logging done in the

on_task_instance_finish

hook doesn’t show up in my single container airflow setup, using a SequentialExecutor. Only the dag run hooks have logs show up. 3. If i wanted to package up my changes to the plugin and run them on a production airflow setup (before making a potential PR), would the best way to do so be via building a wheel for it? I’m still a little confused why i had to install a locally built wheel for the datahub ingestion lib instead of just referencing one from pypi in order for my plugin to work on my local airflow instance.

average-vr-23088

02/29/2024, 5:46 PM

nvm about 2. I see that airflow puts those logs in the task specific logs on disk. Kind of annoying though.

gray-shoe-75895

03/02/2024, 1:09 AM

DatahubLineageBackend is used on older versions of airflow Yes building a wheel would be a good approach. The airflow plugin depends on the main acryl-datahub / metadata-ingestion code, and there were some changes in the airflow plugin that made it require changes to acryl-datahub that weren't in pypi yet. If you try again with acryl-datahub 0.13.0, it'll probably work out of curiosity, what changes are you making to the airflow plugin?

Open in Slack

Previous Next