A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi all, because my organization doesnt support Helm yet, I rendered raw yamls from datahub’s helm chart and used Kustomize to deploy on k8s. Does anyone know how I can get this new configurable profiling working there? <https://github.com/linkedin/datahub/pull/3453>

How are you running ingest right now? Are you using the ingestion-cron or are you exposing gms and running ingest somewhere else?

In both cases, you shouldn’t have to change anything on the helm side to get this working

For the former, you need ingestion-cron to be of version v0.8.16 or higher (correct me if i’m wrong <@U028L1V9BE1> - did this change go in v0.8.16)

For latter, you just need to upgrade your datahub cli and add the settings to your recipe!

I am using the ingestion-cron with image `linkedin/datahub-ingestion:v0.8.16` . But it seems that the PR was merged after. Does that mean I will need to wait for the next release, update the image tag, and the re-deploy?

The other choice would be to use tag “head” which will have the above change!

So you can make sure everything is working

You will have to update the recipe you added to teh configmap accordingly as well

Hi <@U02H4RG7U57>, you need `0.8.16.2` to be able to use the configurable profiling.
```rlanka$ datahub --version
acryl-datahub, version 0.8.16.2``

Cool! I have upgraded to the latest. Let me try it

<@U028L1V9BE1> I am using 0.8.16.2 and ingesting with the following recipe but columns with names like `json_dump` are still getting profiled. Is this a version issue or am I screwing up the regex?
```source:
  type: bigquery
  config:
    project_id: project_id
    env: "DEV"
    profiling:
      enabled: true
      max_workers: 10
    profile_pattern:
      deny:
        - "/json/i"

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"```

Try using `.*json_dump.*` as your deny pattern. The regular expression syntax is that of Python's re.