Hello, could someone reivew the PR. It is that spark lineage result (hdfs/s3) is different from glue lineage result (include s3). spark lineage has partition location, but glue lineage doesn't have partition location. so I add rule. Thank you.

@white-xylophone-3944 does this change add hive table instead of S3 location for upstream lineage?

no, I want to do that. but I think it couldn't in datahub. so I try to link s3 location. but the problem is that s3 location include partition column(location) in spark lineage (e.g. s3://bucket/database/table/*partition=11*/). but there is not partition column in glue lineage (e.g. bucket/database/table/). They doesn't match. I fix it.

got it, so this shows the upstream as s3 location without partition?

It is for that. but I can't know partition pattern so I give user can remove patitions using regexp.

got it, thanks! this will help us a lot