Hello, I am currently trying to ingest multi-project GCP but its only ingest “second_project” which is written as storage_project_id. I want ingest all projects in GCP. What should I?

source:
    type: bigquery
    config:
        project_id: service_acc_project
        storage_project_id: second_project
        credential:
            project_id: service_acc_project
            private_key_id: '${BQ_PRIVATE_KEY_ID2}'
            client_email: <mailto:abc-abc@service.iam.gserviceaccount.com|abc-abc@service.iam.gserviceaccount.com>
            private_key: '${BQ_PRIVATE_KEY2}'
            client_id: '11111111'
        include_tables: true
        include_views: true
        include_table_lineage: true

We have a new beta bigquery connector called

bigquery-beta

which supports this usecase.

Thank for response @dazzling-judge-80093, is it possible to ingest all project in organization without giving project ids? If it is possible can you provide me a example recipe yml?

In bigquery-beta, you don’t have to set the project_id property, and then it will discover all the projects automatically:

source:
  type: bigquery-beta
  config:
    include_table_lineage: true
    include_usage_statistics: true
    profiling:
      enabled: true
      profile_table_level_only: true

sink: