Hello Team,
Is there a way to populate “Documentat...
# troubleshootg
gray-ocean-32209
02/07/2023, 10:19 AMHello Team,
Is there a way to populate “Documentation” information for Airflow DAG at the task level? Similar to the description field Airflow DAGs is used to fill up documentation at the pipeline level.
We use DataHub Airflow lineage plugin for Airflow Datahub integration
Datahub version 0.9.5
Airflow v2.3.4
✅ 1
g
gentle-hamburger-31302
02/07/2023, 1:06 PMAs per
DataHub Airflow lineage plugingentle-hamburger-31302
02/07/2023, 1:12 PMAirflow task support doc, doc_md and other field to set the documentation, I believe we do need a enhancement in plugin to capture the detail from any of these properties
g
gray-ocean-32209
02/07/2023, 3:23 PMThanks for confirming @gentle-hamburger-31302 👍
w
wide-florist-83539
04/11/2023, 2:15 PMWas this feature available @gentle-hamburger-31302? Ingesting the docs_md from airflow as documentation? I submitted a request for this regardless - https://feature-requests.datahubproject.io/p/support-airflow-documentations-as-datahub-documentation
g
gentle-hamburger-31302
04/11/2023, 3:02 PM@green-football-43791 Could you please address this ?
w
wide-florist-83539
04/13/2023, 7:19 PMFollowing up on this ☝️
wide-florist-83539
04/13/2023, 7:51 PMUpdating this request. It looks like datahub pulls in the docs at the pipeline level and not at the task level. Shouldnt this be the case for all tasks in an airflow pipeline?
q
quiet-television-68466
04/27/2023, 4:33 PMLooks like it works for me at the task level?
Copy code
task2 = BashOperator(
task_id="run_data_task2",
doc_md="""
This is what the second data task does.""",
dag=dag,
bash_command="echo 'hello world 2'",)quiet-television-68466
04/27/2023, 4:34 PMWould be great if you could render markdown in Datahub (like you can for other fields), but doesn’t seem like that’s possible now.
Tried to write some markdown for the first query, and it didn’t work.
Copy code
doc_md="""
This is what the first data task does. You can insert a [link](<https://www.youtube.com/watch?v=dQw4w9WgXcQ>).
## You can add a section
you can have content in the subsection
### you can add a subsection
- that subsection can have bullet points
- wow a second bullet point!
"""plus1 1
w
wide-florist-83539
04/28/2023, 12:07 PMAgreed, can this feature be added. Have datahub render the doc_md as markdown in the UI?