Hi,the ingestion metadata(bigquery),multiple table...
# ingestion
f
Hi,the ingestion metadata(bigquery),multiple tables have the same structure but different table names, such as user_ 1,user_ 2,Why is there only table user in the final dataset。Is there any logic for merging? If so, can it be canceled @dazzling-judge-80093
d
well, that is one way we try to support sharded tables in Bigquery, and currently, you can’t disable it. Let us figure out how we can support sharded table and this usecase as well.
f
Can I understand it this way? This situation is treated as a partitioned table. But what is the matching rule of this partitioned table? Is it treated as a partitioned table as long as the table structure is the same
Hi, I use “sharded_table_pattern”,an error occurred @dazzling-judge-80093
Copy code
source:
  type: bigquery
  config:
    project_id: ${SOCIAL_INSIGHTS_BIGQUERY_ID}
    credential:
      project_id: ${SOCIAL_INSIGHTS_BIGQUERY_ID}
      private_key_id: ${SOCIAL_INSIGHTS_PRIVATE_KEY_ID}
      private_key: ${SOCIAL_INSIGHTS_PRIVATE_KEY}
      client_email: ${SOCIAL_INSIGHTS_CLIENT_EMAIL}
      client_id: ${SOCIAL_INSIGHTS_CLIENT_ID}
    sharded_table_pattern: ((.+)[_$])?(\d{4,10})$
    
sink: 
  type: datahub-rest
  config:
    server: ${DATAHUB_GMS}
    token: ${TOKEN}
``````