Mitch Zink
09/25/2025, 12:28 PMkapa.ai
09/25/2025, 12:28 PMkapa.ai
09/25/2025, 12:28 PMall_columns_anomalies) across a very large number of columns (e.g., 50,000) in your dbt project is likely to cause performance and resource issues. There are several reports from users experiencing long runtimes, timeouts, and errors (including on-run-end hook failures and query complexity/resource errors) when working with high column counts in Elementary, especially on BigQuery. For example, one user with ~85,000 columns reported that dbt run -s elementary took over 30 minutes, and dbt test errored out due to query complexity and resource limits. The error messages included "Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex" and issues with query length limits. These problems are directly related to the scale of the metadata and the number of tests Elementary tries to generate and run for each column in your project [see discussion here](https://slack.com/archives/C02CTC89LAX/p1711672843688409). Additionally, there are known issues with the dbt_columns model and related metadata models failing or taking a long time to materialize when there are a large number of schemas or columns in the project [see related issue](https://github.com/elementary-data/elementary/issues/945). While the specific error about not being able to open a PR is not detailed in the knowledge sources, it is reasonable to expect failures or timeouts when attempting to run such a high volume of column anomaly tests. The recommended approach in these situations is to: - Limit the scope of the tests (e.g., run column_anomalies only on selected columns rather than all columns). - Use exclusion patterns to avoid problematic or unnecessary columns. - Consider splitting tests across multiple runs or increasing available resources if possible. If you need to monitor all columns, you may need to implement workarounds or optimizations, as running everything at once is likely to hit system or platform limits [see performance discussion](https://slack.com/archives/C02CTC89LAX/p1658915396893639). In summary, yes, this behavior is expected given the high volume, and you may need to adjust your approach to avoid these issues.Noya Offer
09/25/2025, 2:06 PM