Hi，Ingest bigquery and add sharded_ table_ Pattern and the default value is used. An error is reported. Has anyone ever encountered it。@dazzling-judge-80093

source:
  type: bigquery
  config:
    project_id: ${SOCIAL_INSIGHTS_BIGQUERY_ID}
    credential:
      project_id: ${SOCIAL_INSIGHTS_BIGQUERY_ID}
      private_key_id: ${SOCIAL_INSIGHTS_PRIVATE_KEY_ID}
      private_key: ${SOCIAL_INSIGHTS_PRIVATE_KEY}
      client_email: ${SOCIAL_INSIGHTS_CLIENT_EMAIL}
      client_id: ${SOCIAL_INSIGHTS_CLIENT_ID}
    sharded_table_pattern: ((.+)[_$])?(\d{4,10})$
    
sink: 
  type: datahub-rest
  config:
    server: ${DATAHUB_GMS}
    token: ${TOKEN}

Hi @full-chef-85630, could you share the error that is reported?

@helpful-optician-78938

failures={'cube': ['Tables error: first argument must be string or compiled pattern'], 
'dedup': ['Tables error: first argument must be string or compiled pattern'], 
'logic': ['Tables error: first argument must be string or compiled pattern', 
'Views error: first argument must be string or compiled pattern'], 
'operation': ['Tables error: first argument must be string or compiled pattern'], 
'origin': ['Tables error: first argument must be string or compiled pattern'],
'test': ['Tables error: first argument must be string or compiled pattern'], 
'views': ['Tables error: first argument must be string or compiled pattern',
 'Views error: first argument must be string or compiled pattern']

I'll look into this and get back to you soon

@helpful-optician-78938 still not ，this parameter is a compilation problem

from datahub.ingestion.run.pipeline import Pipeline

pip =Pipeline.create(config_dict={
	'source': {
		'type': 'bigquery',
		'config': {
			'project_id': '',
			'sharded_table_pattern': '((.+)[_$])?(\d{4,10})$',
			'credential': {
				'project_id': '',
				'private_key_id': '',
				'private_key': '',
				'client_email': '',
				'client_id': ''
			}
		}
	},
	'sink': {
		'type': 'datahub-rest',
		'config': {
			'server': '',
			'token': ''
		}
	}
})

pip.run()
pip.raise_from_status()

Looks like you are trying to supply the default value here. Can you try excluding it altogether from the config, so that the default gets used? cc: @dazzling-judge-80093

I want to implement custom, but I only use the default value after an error is reported @dazzling-judge-80093@loud-island-88694

Hi @full-chef-85630, it was not working for you because of a bug. Here is the PR with the fix. Once it is merged, you can try on the latest master.

thank you! I have identified the problem

We jus cut a new release which contains this fix it should show up on pypi soon

thank you