When you run a column anomaly test in Elementary with a

timestamp_column

configured, the test splits your data into time buckets (e.g., daily, if using a daily bucket). For each run, it calculates metrics for each bucket defined by

bucket_start

and

bucket_end

, and appends a new row for each bucket to the

data_monitoring_metrics

table. This is by design: the table is intended to store a historical record of metrics for each time bucket, not just the latest value. This approach allows Elementary to perform anomaly detection over time by comparing metrics across different buckets. As a result, running the test multiple times will continue to append new metric rows for new time buckets, rather than merging or overriding previous entries. This is necessary for accurate anomaly detection, but it does mean that the

data_monitoring_metrics

table can grow rapidly, especially if tests are run frequently or on large datasets with fine-grained time buckets. If you notice rapid growth, it's a consequence of this historical tracking mechanism. There is no mention in the knowledge sources of automatic merging or deduplication for the same bucket in repeated runs; each run appends new data for the relevant time buckets based on the current state of the data and the test configuration. This behavior is confirmed in several discussions about how metrics are aggregated and stored for each bucket and run, especially when a

timestamp_column

is set, and how the system is designed to keep historical data for anomaly detection purposes [How timestamp_column affects bucketing and metrics](https://slack.com/archives/C02CTC89LAX/p1694016833712689), [Data aggregation and storage in data_monitoring_metrics](https://slack.com/archives/C02CTC89LAX/p1666376716952649). If you have concerns about table growth, you may need to implement your own data retention or cleanup strategy, as the knowledge sources do not mention built-in retention policies.

Based on the available knowledge sources, the ID field in the

data_monitoring_metrics

table is generated as a hash of several fields. In one example, the ID is created using the following fields: -

full_table_name

-

column_name

-

metric_name

-

bucket_end

- (sometimes)

metric_properties

For example, the ID is generated as follows:

sql to_hex(   md5(     cast(       coalesce(cast(full_table_name as string), '') || '-' ||       coalesce(cast(column_name as string), '') || '-' ||       coalesce(cast(metric_name as string), '') || '-' ||       coalesce(cast(bucket_end as string), '') || '-' ||       coalesce(cast(metric_properties as string), '')     as STRING)   ) ) as id

[source](https://slack.com/archives/C02CTC89LAX/p1682055726978159) In other examples, the ID is generated without

metric_properties

, but always includes

full_table_name

,

column_name

,

metric_name

, and

bucket_end

(or sometimes

bucket_start

):

sql to_hex(md5(cast(   coalesce(cast(full_table_name as string), '') || '-' ||   coalesce(cast(column_name as string), '') || '-' ||   coalesce(cast(metric_name as string), '') || '-' ||   coalesce(cast(bucket_end as string), '') as string))) as id

[source](https://slack.com/archives/C02CTC89LAX/p1657883114555219) Key points: - The ID is not always generated with

bucket_start

, but rather with

bucket_end

. - In some cases,

metric_properties

is also included in the hash. - The ID is intended to uniquely identify a metric for a specific table, column, metric type, and time bucket. Conclusion: If you choose to deduplicate using the ID field, it is generally safe as long as you are aware that the uniqueness is based on the combination of

full_table_name

,

column_name

,

metric_name

,

bucket_end

, and sometimes

metric_properties

. If your use case requires deduplication by

bucket_start

, you should verify that your version of Elementary uses

bucket_start

in the ID generation, as the knowledge sources show that

bucket_end

is more commonly used. If you want to be absolutely certain, you should check your specific implementation or version, as there may be slight variations in how the ID is constructed. The knowledge sources do not provide a definitive answer for all versions, but the above reflects the most common patterns found in the shared queries.

The knowledge sources indicate that the

data_monitoring_metrics

table in Elementary stores historical metric data for each test run, with each run appending new rows for the relevant time buckets. The fields

updated_at

and

created_at

are both present in this table. -

created_at

typically records when the row was first inserted into the table. -

updated_at

records the time the metric was last updated. However, since each test run appends new rows (rather than updating existing ones), the practical distinction between these two fields is minimal in this context. Both will often have similar or identical values, reflecting the time of insertion for that particular metric record. The presence of both fields may be a result of standard data modeling practices or to support future scenarios where updates (rather than inserts) might occur, but the knowledge sources do not provide a specific reason for having both fields in the current append-only workflow. In summary, with the current append-only behavior, there is little functional difference between

created_at

and

updated_at

in the

data_monitoring_metrics

table, and the knowledge sources do not explicitly clarify a unique purpose for each in this context.