https://datahubproject.io logo
c

cool-vr-73109

07/26/2022, 9:26 AM
Hi team, I tried s3 ingestion file lineage from cli and I could see a new source got ingested with lineage tab enabled and didn't enabled by existing ingestion lineage tab. Which part of s3 ingestion name should give as entity name in file lineage ingestion? Just dataset name Or bucketname/folder/S3_filename?
c

careful-pilot-86309

07/26/2022, 10:49 AM
Dataset name can be controlled by path_spec.table_name config along with user defined varibales in path_spec.include. Please refer here for examples: https://datahubproject.io/docs/quickstart#valid-path_specsinclude
In the examples {dept} is the user defined variable. Whenever creating table name config you can use them
m

microscopic-room-90690

01/17/2023, 10:38 AM
@careful-pilot-86309 Does user defined variable only can be shown in datahub as Folder/Container?
3 Views