Hi,the ingestion metadata(bigquery),multiple table...
# ingestionf
full-chef-85630
09/05/2022, 1:35 PMHi,the ingestion metadata(bigquery),multiple tables have the same structure but different table names, such as user_ 1，user_ 2，Why is there only table user in the final dataset。Is there any logic for merging? If so, can it be canceled @dazzling-judge-80093
d
dazzling-judge-80093
09/05/2022, 3:53 PMwell, that is one way we try to support sharded tables in Bigquery, and currently, you can’t disable it.
Let us figure out how we can support sharded table and this usecase as well.
f
full-chef-85630
09/05/2022, 11:44 PMCan I understand it this way? This situation is treated as a partitioned table. But what is the matching rule of this partitioned table? Is it treated as a partitioned table as long as the table structure is the same
full-chef-85630
09/06/2022, 10:00 AMHi, I use “sharded_table_pattern”,an error occurred @dazzling-judge-80093
Copy code
source:
type: bigquery
config:
project_id: ${SOCIAL_INSIGHTS_BIGQUERY_ID}
credential:
project_id: ${SOCIAL_INSIGHTS_BIGQUERY_ID}
private_key_id: ${SOCIAL_INSIGHTS_PRIVATE_KEY_ID}
private_key: ${SOCIAL_INSIGHTS_PRIVATE_KEY}
client_email: ${SOCIAL_INSIGHTS_CLIENT_EMAIL}
client_id: ${SOCIAL_INSIGHTS_CLIENT_ID}
sharded_table_pattern: ((.+)[_$])?(\d{4,10})$
sink:
type: datahub-rest
config:
server: ${DATAHUB_GMS}
token: ${TOKEN}