A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

If so, can someone lmk what you used for configurations in the recipe.yml file?

Configure recipe something like:
`source:`
  `type: bigquery`
  `config:`
    `# Coordinates`
    `project_id: calm-pagoda-323403`



    `# `schema_pattern` for BQ Datasets`
    `schema_pattern:`
      `allow:`
        `- bigquery_demo`
`#        - usage`

    `table_pattern:`
      `allow:`
        `- .*customer`
      `deny:`
        `- .*customer_orders`
        `- .*customer_derived`

    `stateful_ingestion:`
      `enabled: false`
      `state_provider:`
       `type: "datahub" # default value`
      `# This section is needed if the pipeline-level `datahub_api` is not configured.`
       `config:  # default value`
          `datahub_api:`
              `server: "<http://localhost:8080>"`

Before executing ingestion set your creds with command:
`export GOOGLE_APPLICATION_CREDENTIALS=&lt;path/to/bigquery-token.json&gt;