This message was deleted.

Hi Karishma, profiling can take an extremely long time depending on your configuration and size of data. It may make sense to only profile subsets of all of the available datasets in batches to be able to see if it is correctly processing if your cron job is not outputting any logs.

@hundreds-photographer-13496 ^ can you please clarify what configuration is needed for incremental profiling?

@creamy-van-28626 which datahub cli version are you using ? In datahub 0.8.39 release, selective profiling is enabled by default. Only tables that have been updated within the prior N (default 1) days are profiled. You can update value for N using config

profiling.profile_if_updated_since_days