A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi All,

Currently the <https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub/charts/datahub-ingestion-cron|K8s ingestion cron> chart component creates a CronJob resource hardcoded to launch the metadata ingestion framework for a given configuration. This is not very flexible in cases where you may want to customize the output of the metadata ingestion framework or launch something else altogether.

I found this need myself when trying to enrich the output of the ingestion framework before sending the MCEs to DataHub. In order to do so I generalized the ingestion chart to allow the possibility to define a generic shell command with custom logic through a bash script. If anyone think this is useful let me know and I'll open a PR. cc <@U01GZEETMEZ> <@UV0M2EB8Q>

Thanks for the details Pedro. Cc <@U01NN9CTQ01> 

Are you saying that the shell command intercepts metadata prior to it being sent to DH?

<@U01L2QD243Z>: have you taken a look at the Transformers support recently added? <https://datahubproject.io/docs/metadata-ingestion#transformations>

My bash script happens to do that but you can run anything.

<@UV0M2EB8Q> I think transformers were not ready when I did this work

I'm just asking whether you guys are interested in a more generalized ingestion cron job. I can also submit the PR and you can let me know :slightly_smiling_face:

Definitely open the PR and we can discuss there! Now that transformers are supported, would prefer that be the primary way to customize ingestion post-processing. But there might be other reasons to customize the ingestion script beyond that as well as you said.