Has anyone tried ingesting from bigquery? I was testing it out and cant make the pattern to work. For starters, I am trying to ingest and profile one specific table. I tried this but it ingest all tables in db1.

source:
  type: bigquery
  config:
    project_id: "su-project1"
    schema_pattern: 
        allow: 
           - "db1"
    table_pattern: 
        allow:
           - "db1.table1"   
    profiling: {
      enabled: true,
    }   
    profile_pattern:
        allow:
           - "db1.table1" 
      
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

I tried removing the schema pattern and retain the table pattern but it ingest all database in the project. Are there other options I need to set?

@loud-vase-59377 Can you verify if you are ingesting tables or views? You can do that by changing the values of

include_tables

and

include_views

in the yml config (more here). Based on that, apply either the table pattern or the view pattern with the syntax

schema_name.entity_name

(entity being table or view).

Thanks! It's the views that I am seeing in the logs.

Were you able to try the pattern on it?

@witty-state-99511 I am facing similar issues (in bigquery). I want to ingest both table and views. I try to deny some schemas using a regex pattern, but it still ingests those schemas (or datasets). Also another issue is while writing the regex pattern, doing something like this

r"1234$"

gives an error

Unbounded variable

due to

$

symbol, so we need to escape it, but then regex pattern `r"1234\$"`won't work.