Hello everyone. I'm excited about the potential of DataHub. I successfully ran ingestion from glue but find no lineage for most of our jobs. The curious thing is that under Platforms, I see two boxes: Glue with 315 objects and S3 with 4. Only the ones listed as S3 give me lineage. None of the Glue jobs do. All our sources and targets are in S3. Is there a setting or config required to extract lineage from Glue jobs? Any hints would be apprecaited.

Hi @important-camera-38424! Looking into this on our side.. Should have an update soon

Thanks @big-carpet-38439. @important-camera-38424 will update this thread soon.

Hi @important-camera-38424! Looking into this now. What kind of lineage are you expecting? When you mention that you see lineage in S3, can you send a screenshot of what that looks like?

Thanks for reaching out. Here's an example of what I'd expect.

Interesting… does there seem to be a difference between the jobs that have lineage vs. those that don’t?

They are custom PySpark jobs

Ah so this is a known caveat with Glue – their API for getting lineage only works if you’ve constructed the script using the GUI. As far as we know there’s no way around this limitation

Ah. OK. Bummer. And you're sure there is no way to adjust or add to our PySpark jobs to get lineage?

Don’t think it’s possible through Glue unfortunately

That would be cool, although it's not clear to me how that would work. There is this: hadoop - What is Lineage In Spark? - Stack Overflow

Oh that’s unexpected then, can you share what kind of job you created?

It's a very simple job that moves data from one S3 bucket to another and drops a column along the way.

Following this thread as I also work with Glue + Custom PySpark scripts 🙂

Cc @chilly-holiday-80781

Update: A fix was released to pull lineage but only from annotation. The annotation is no longer generated in Glue Studio unless you toggle to "Generate Classic Script". For custom PySpark code, you would have to write up the annotation manually. Here's the update made by a DataHub team member: https://github.com/linkedin/datahub/pull/3555/files

Hi @best-carpenter-16586 thanks for the update! i did a quick test to see if this would work with my custom script, i only used 2 annotations: 1st: to annotate source bucket

## @type: DataSource
## @args: [connection_type = "s3", format = "parquet", connection_options = {"paths": ["<s3://xxx/xxx/>"], "recurse":True}, transformation_ctx = "DataSource0"]
## @return: DataSource0
## @inputs: []

2nd: to annotate my target bucket

## @type: DataSink
## @args: [format_options = {"compression": "snappy"}, connection_type = "s3", format = "parquet", connection_options = {"path": "<s3://yyy/yyy>"}, transformation_ctx = "DataSink0"]
## @return: DataSink0
## @inputs: [frame = DataSource0]

the glue script looks like this