Hi Everyone, Trying to install S3 source pip install 'acryl-datahub[s3]==0.9.3' --constraint "${constraint_url}" ( constraint URL is for Airflow https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt ) But unable to do so, due to pyspark dependency conflict, so Raised a bug https://github.com/datahub-project/datahub/issues/6852 Error:

ERROR: Cannot install acryl-datahub[s3]==0.9.4 because these package versions have conflicting dependencies.

The conflict is caused by:
    acryl-datahub[s3] 0.9.4 depends on pyspark==3.0.3; extra == "s3"
    The user requested (constraint) pyspark==3.2.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Thanks for reporting! @gray-shoe-75895 Are you able to take a look?

@handsome-football-66174 It looks like you’re using Airflow 1.10.12 - we actually dropped support for airflow 1.x a while back. You can use an older version of our client library as per the note here https://datahubproject.io/docs/lineage/airflow/

I believe it's the same for delta lake, I saw it's downgrading my pyspark version to 3.0.3 ( from 3.3.0)

@thankful-fireman-70616 I suspect that’s because we’ve pinned our pyspark version here https://github.com/datahub-project/datahub/blob/a1970d2dcee1afbf5c5de06c175166ccc04e1c8d/metadata-ingestion/setup.py#L218-L219. I don’t think there’s a specific reason that we need to pin it, so we should be able to loosen that requirement

@gray-shoe-75895 We have changed the airflow version to Airflow 2.3.2. But we are still getting same error. ERROR: Cannot install acryl-datahub[s3]==0.9.3 because these package versions have conflicting dependencies. The conflict is caused by: acryl-datahub[s3] 0.9.3 depends on pyspark==3.0.3; extra == "s3" The user requested (constraint) pyspark==3.2.1 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict

Yes, that's pretty much causing the issue. Also, you can check the comparability of delta-lake as well while bumping up pyspark to the latest...

loosening the deps here https://github.com/datahub-project/datahub/pull/6908