Hello, everyone! I have a question about the lineage of the datahub. I searched the information of the lineage and found out how to draw lineage automatically with airflow integration. But I'd like to know more how to automatically set lineage with logs etc. Especitally, if data is flowing in EAI, ETL, etc, how can I draw the lineages at once?? Is there a way to draw a non-manual lineage? Thanks in advance for your help!!!

As far as I know, Datahub currently provides Spark and Ariflow lineage ingestion tools. If there are some customized data processing tasks/tools in your architecture, you may need to report these information to the Datahub yourself, so as to render blood in the datahub. The details of the report may depend on your tools. For example, we have developed the corresponding hook to analyze the relationship of the data task and automatically report it to the Datahub.