Maybe a dumb question - how can I properly manage ...
# troubleshoot
f
Maybe a dumb question - how can I properly manage the path for a dataset when emitting a change to the metadata to datahub? (Using the python approach) For example, I have some data in s3 buckets. In the script, I loop over the objects in said bucket, and, using a
MetadataChangeProposalWrapper
, emit changes to add new custom properties. But the problem is, when I give the entityUrn the name (via
make_dataset_urn
), it treats the filepath both as the filepath and name of the object. So instead of ending up with a
<env>/<folderpath>/<obj_name>
in datahub, I end up with that path, but the object name itself includes the folder path as if it was part of the object name itself.
✅ 1
a
Hi Jon, you may need to look into a transformer to change the name on ingest https://datahubproject.io/docs/metadata-ingestion/docs/transformer/intro/
f
From my testing yesterday, I may have made a mistake in the original script that caused this problem. I will keep this in mind if needed. Thank you!