I'm trying to setup the generic column anomalies t...
# ask-elementary-ai
m
I'm trying to setup the generic column anomalies test for everything in my dbt project, it's 50k cols. It loads for a while, then fails and says it couldn't open a PR. Is this expected given the high volume?
k
A message from kapa.ai
Running the generic column anomalies test (such as
all_columns_anomalies
) across a very large number of columns (e.g., 50,000) in your dbt project is likely to cause performance and resource issues. There are several reports from users experiencing long runtimes, timeouts, and errors (including on-run-end hook failures and query complexity/resource errors) when working with high column counts in Elementary, especially on BigQuery. For example, one user with ~85,000 columns reported that
dbt run -s elementary
took over 30 minutes, and
dbt test
errored out due to query complexity and resource limits. The error messages included "Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex" and issues with query length limits. These problems are directly related to the scale of the metadata and the number of tests Elementary tries to generate and run for each column in your project [see discussion here](https://slack.com/archives/C02CTC89LAX/p1711672843688409). Additionally, there are known issues with the
dbt_columns
model and related metadata models failing or taking a long time to materialize when there are a large number of schemas or columns in the project [see related issue](https://github.com/elementary-data/elementary/issues/945). While the specific error about not being able to open a PR is not detailed in the knowledge sources, it is reasonable to expect failures or timeouts when attempting to run such a high volume of column anomaly tests. The recommended approach in these situations is to: - Limit the scope of the tests (e.g., run
column_anomalies
only on selected columns rather than all columns). - Use exclusion patterns to avoid problematic or unnecessary columns. - Consider splitting tests across multiple runs or increasing available resources if possible. If you need to monitor all columns, you may need to implement workarounds or optimizations, as running everything at once is likely to hit system or platform limits [see performance discussion](https://slack.com/archives/C02CTC89LAX/p1658915396893639). In summary, yes, this behavior is expected given the high volume, and you may need to adjust your approach to avoid these issues.
n
Hey @Mitch Zink ! Sending you a private message 🙌