Hi team, I tried s3 ingestion file lineage from cl...
# ingestionc
cool-vr-73109
07/26/2022, 9:26 AMHi team, I tried s3 ingestion file lineage from cli and I could see a new source got ingested with lineage tab enabled and didn't enabled by existing ingestion lineage tab. Which part of s3 ingestion name should give as entity name in file lineage ingestion? Just dataset name Or bucketname/folder/S3_filename?
c
careful-pilot-86309
07/26/2022, 10:49 AMDataset name can be controlled by path_spec.table_name config along with user defined varibales in path_spec.include. Please refer here for examples: https://datahubproject.io/docs/quickstart#valid-path_specsinclude
careful-pilot-86309
07/26/2022, 10:50 AMIn the examples {dept} is the user defined variable. Whenever creating table name config you can use them
m
microscopic-room-90690
01/17/2023, 10:38 AM@careful-pilot-86309 Does user defined variable only can be shown in datahub as Folder/Container?
3 Views