Hello Team We are experimenting with testing Airfl...
# troubleshoot
g
Hello Team We are experimenting with testing Airflow datahub integrations (Airflow lineage backend) with datahub quickstart and datahub-airflow docker https://datahubproject.io/docs/docker/airflow/local_airflow
Copy code
[lineage]
backend = datahub_provider.lineage.datahub.DatahubLineageBackend
datahub_kwargs = {
    "datahub_conn_id": "datahub_rest_default",
    "cluster": "local_airflow",
    "capture_ownership_info": true,
    "capture_tags_info": true,
    "capture_executions": true,
    "graceful_exceptions": true }
To see the information about the runs history of airflow tasks in datahub added
"capture_executions": true
whenever we add this option `"capture_executions": true`` and try to initialize airflow with cmd
docker-compose up airflow-init
it fails with
Copy code
....
datahub-airflow-airflow-init-1  |     _backend = get_backend()
datahub-airflow-airflow-init-1  |   File "/home/airflow/.local/lib/python3.9/site-packages/airflow/lineage/__init__.py", line 61, in get_backend
datahub-airflow-airflow-init-1  |     return clazz()
datahub-airflow-airflow-init-1  |   File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/lineage/datahub.py", line 64, in __init__
datahub-airflow-airflow-init-1  |     _ = get_lineage_config()
datahub-airflow-airflow-init-1  |   File "/home/airflow/.local/lib/python3.9/site-packages/datahub_provider/lineage/datahub.py", line 35, in get_lineage_config
datahub-airflow-airflow-init-1  |     return DatahubLineageConfig.parse_obj(kwargs)
datahub-airflow-airflow-init-1  |   File "pydantic/main.py", line 511, in pydantic.main.BaseModel.parse_obj
datahub-airflow-airflow-init-1  |   File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__
datahub-airflow-airflow-init-1  | pydantic.error_wrappers.ValidationError: 1 validation error for DatahubLineageConfig
datahub-airflow-airflow-init-1  | capture_executions
I’m running
acryldata/airflow-datahub:latest
image is ’`capture_executions`, is not supported?
b
Investigating this - it appears that whatever version is getting pulled indeed does not have this - It also doesn't look like we are actively publishing this image with newer version of the DataHub CLI. My guess is that this docker container does not contain the latest version of DataHub CLI (acryl-datahub) which explains why you're seeing this