Hello, I'd like to apply datahub for my BigQuery. ...
# ingestiona
adorable-flower-19656
02/15/2022, 3:08 AMHello, I'd like to apply datahub for my BigQuery. It has about 7000 tables.
When I succeed ingestion, it takes about 1~2 hours.
But sometimes the ingestion fails with "NoSuchTableError".
It seems to have something to do with the table being dropped during ingestion execution. But I'm not sure.
The creation/deletion of the tables frequently occur.
My questions are,
1. If there is a table that is deleted during ingestion, do I fail to ingest?
2. If so, can I ignore the deleted tables and complete the ingestion?
s
square-activity-64562
02/15/2022, 7:38 AMIf the temporary tables are in a different schema you can use a schema deny pattern to filter those out. If they have some prefix or suffix then you can use table apply/deny patterns.
square-activity-64562
02/15/2022, 7:38 AMHaving said that it should not be failing the process completely in case table fails. Can you post a stacktrace in the thread here?
a
adorable-flower-19656
02/15/2022, 8:02 AMadorable-flower-19656
02/15/2022, 8:06 AM@square-activity-64562, Thanks for your reply.
I posted my stacktrace log from UI ingestion.
I edited actual table name in the log.
adorable-flower-19656
02/16/2022, 9:18 AM@square-activity-64562, Hi. I found an similar issue. https://github.com/linkedin/datahub/issues/2627
Can I register new issue for this BQ ingestion problem?
s
square-activity-64562
02/16/2022, 9:19 AMThe link that you gave is not working for me. Not sure why. Yes, please create a issue for BigQuery error
a
adorable-flower-19656
02/16/2022, 9:24 AMI'm sorry there is a typo. I will create an issue. Thanks!
adorable-flower-19656
02/17/2022, 8:01 AM@square-activity-64562, I created github issue.
If you need more description about the issue, please let me know.
https://github.com/linkedin/datahub/issues/4165