Hello All, i have added glue metadata using cli. I can see the datasets on ui now. But not lineage. Any idea how to add lineage for glue source

@lemon-engine-23512 There are few configurations, which can be used to control the lineage. I hope you have taken look at this. Can you share your recipe to confirm if any change is needed over there.

Thank you for responding @careful-pilot-86309 . I just used this recipi source: type: glue config: # Coordinates aws_region: "my-aws-region" sink: # sink configs

Please set

emit_s3_lineage

to true

Will so that and try. The previous recipi has already loaded the metadata. Do i run the recipi again?

no need to remove it will get overwritten

Thank you @careful-pilot-86309 . One more thing. I have set up datahub quickstart on aws ec2 machine. Usually i stop the instance at the end of day as its not in use. But when i start it again, datahub service is not up. In containers i can only see datahub actions. And ui not accessible. How do i start the services again

Youbcan do datahub docker quickstart

@careful-pilot-86309 it worked thanks. Also how i do lineage from glue properties. It holds base tables and pipeline name info, can we build lineage with this using datahub

Have you checked this This source actually pulls up lineage automatically based on config parameters.

Yeah i did, and used the recipi as mentioned. But it was only able to bring s3 path into lineage. Not rhe base tables or piplines

@gentle-hamburger-31302 can you please take it forward with glue details?

@gentle-hamburger-31302 would appreciate any help. So i have added glue to my datahub and now i see the datasets and lineage(s3 location). But i want the lineage to display the upstream base tables as well. We have added a custom property in glue with base tables and pipeline name info. Will glue be able to build lineage on this? Or can we do some custom lineage to get this ?