<@U088WCP3B4P> While monitoring query execution i...
# ask-elementary-ai
k
@kapa.ai While monitoring query execution in Snowflake, I noticed a particular query consuming a noticeable amount of time and compute:
Copy code
SELECT COUNT(*)
FROM {database}.INFORMATION_SCHEMA.schemata
WHERE UPPER(schema_name) = UPPER('elementary_schema')
  AND UPPER(catalog_name) = UPPER('elementary_database')
After investigating, I found this query only runs on the database where I enabled the Elementary package. I'd like to better understand what this query is doing and how it's being used within the Elementary OSS framework. • Is this part of a routine metadata check or schema validation? • Is it expected to run frequently? • Are there any best practices for minimizing its impact?
k
A message from kapa.ai
The query you observed:
Copy code
sql SELECT COUNT(*) FROM {database}.INFORMATION_SCHEMA.schemata WHERE UPPER(schema_name) = UPPER('elementary_schema')   AND UPPER(catalog_name) = UPPER('elementary_database')
is indeed part of the Elementary OSS dbt package's internal operations. Here’s how it fits into the framework and what you should know: ### Purpose of the Query - Routine Metadata Check / Schema Validation: This query is used by Elementary to validate the existence of specific schemas and databases before running certain models, such as
dbt_columns
,
filtered_information_schema_columns
, and
filtered_information_schema_tables
. It ensures that the schema Elementary expects to interact with actually exists in your Snowflake environment. This is a safeguard to prevent errors when querying metadata or building views that depend on schema presence. [See discussion and query examples](https://slack.com/archives/C02CTC89LAX/p1684251047961949) ### Frequency of Execution - How Often Does It Run? The query is executed as part of the build process for certain Elementary models. In earlier versions, it was run for every model or source, which could result in a large number of queries and increased runtime, especially in projects with many models or sources. Later improvements aimed to reduce this by checking each schema only once per run, but the query is still expected to run at least once per relevant schema during each dbt invocation that builds or refreshes these models. [Explanation of the implementation and its impact](https://slack.com/archives/C02CTC89LAX/p1684251047961949) ### Best Practices to Minimize Impact - Disable Heavy Models (if needed): If you find that these metadata checks are causing significant performance issues, you can disable the specific models responsible. For example, in your
dbt_project.yml
, you can add:
Copy code
yaml   elementary:     edr:       dbt_artifacts:         dbt_columns:           +enabled: false       metadata_store:         filtered_information_schema_columns:           +enabled: false         filtered_information_schema_tables:           +enabled: false
Disabling these models will limit some features (such as schema change tests), but can help reduce compute usage if that's a concern. [See guidance on disabling models](https://slack.com/archives/C02CTC89LAX/p1693840882682879) - Leverage Caching (dbt 1.4.0+): If you are using dbt version 1.4.0 or above, Elementary leverages a metadata cache, so these checks and updates are minimized after the initial run. Upgrading to a supported dbt version is recommended for better performance. [Performance impact and caching details](https://docs.elementary-data.com/dbt/on-run-end_hooks) - Selective Enabling: You can selectively enable or disable artifact collection and model builds based on your monitoring needs. However, be aware that disabling some features may reduce the observability coverage provided by Elementary. ### Summary - The query is a schema existence check, used for metadata validation by Elementary. - It is expected to run during model builds, but frequency can be reduced with newer versions and proper configuration. - You can disable specific models or leverage dbt's caching to minimize impact, but this may limit some features. If you need more granular control or are experiencing persistent performance issues, consider reaching out to the Elementary community or support for tailored advice. Let me know if you need more details or help with configuration!