Hello everyone! I'm a huge fan of elementary and b...
# community-support
d
Hello everyone! I'm a huge fan of elementary and been using introducing it in three companies so far. Lately the result queries in the output of "edr report" don't work anymore. It complains that some tmp tables don't exist. Any idea someone? I directly execute the statement I get from the UI.
maybe it has something to do with edr not having recent package versions in its internal dbt project ?
n
Hey @Daniel Kaiser 👋🏼 Do you get an error while running the command? Where do you run the dbt project and the edr command (your local env or somewhere else)?
d
Hey @Noy Arie! Thanks for the help. It has an anomaly-error in an unrelated test but no error otherwise when running. It's running inside a docker container on amazon deployed through serverless. The dockerfile uses
<http://ghcr.io/dbt-labs/dbt-core:1.7.0|ghcr.io/dbt-labs/dbt-core:1.7.0>
as the base image.and installs edr using pip.
image.png,image.png
n
Can you try running it with dbt version 1.6?
d
Yeah, I'll run it now with 1.6 to make sure. Yesterday it also had the same bug with 1.6 when I tried
Hey @Noy Arie ... just ran it with 1.6 and it's the same issue.
I can give you an example for a column anomaly:
Copy code
select * from (
    -- depends_on: chm_core_prod.elementary.monitors_runs
    -- depends_on: chm_core_prod.elementary.data_monitoring_metrics
    -- depends_on: chm_core_prod.elementary.dbt_run_results
        
        

        
        
        
        

        
        
        
        

        
        


        
        
        
        

        

        
        
        

        
        

        
        
        

        
        
        


        with anomaly_scores as (
        select
            id,
            metric_id,
            test_execution_id,
            test_unique_id,
            detected_at,
            full_table_name,
            column_name,
            metric_name,
            anomaly_score,
            anomaly_score_threshold,
            anomalous_value,
            bucket_start,
            bucket_end,
            bucket_seasonality,
            metric_value,
            min_metric_value,
            max_metric_value,
            training_avg,
            training_stddev,
            training_set_size,
            training_start,
            training_end,
            dimension,
            dimension_value,
            
    case
        when dimension is not null then 
    'The last ' || metric_name || ' value for dimension ' || dimension || ' - ' ||
    case when dimension_value is null then 'NULL' else dimension_value end || ' is ' || round(cast(metric_value as numeric(28,6)), 3) ||
    '. The average for this metric is ' || round(cast(training_avg as numeric(28,6)), 3) || '.'

        when metric_name = 'freshness' then 
    'Last update was at ' || anomalous_value || ', ' || abs(round(cast(metric_value/3600 as numeric(28,6)), 2)) || ' hours ago. Usually the table is updated within ' || abs(round(cast(training_avg/3600 as numeric(28,6)), 2)) || ' hours.'

        when column_name is null then 
    'The last ' || metric_name || ' value is ' || round(cast(metric_value as numeric(28,6)), 3) ||
    '. The average for this metric is ' || round(cast(training_avg as numeric(28,6)), 3) || '.'

        when column_name is not null then 
    'In column ' || column_name || ', the last ' || metric_name || ' value is ' || round(cast(metric_value as numeric(28,6)), 3) ||
    '. The average for this metric is ' || round(cast(training_avg as numeric(28,6)), 3) || '.'

        else null
    end as anomaly_description
,
            max(bucket_end) over (partition by test_execution_id) as max_bucket_end
        from chm_core_prod.elementary."test_bdf5920709_elementary_column_anomalies_page_views_average__ENGAGED_TIME_IN_S__day_of_week__3__day__1__anomaly_scores__tmp_20231116024747364356"
      ),
      anomaly_scores_with_is_anomalous as (
        select
          *,
case when
            anomaly_score is not null and
            (
              
  (
    metric_value = 0 and 
    
      1 = 2
    
  )
 or
              (
                case when metric_name IN 
        ( 'freshness' ,  'event_freshness'  )
 then
            anomaly_score > 3
    else
        
        abs(anomaly_score) > 3
    

     end and
                
      (1 = 1)
  

  and

  
      (1 = 1)
  
              )
            ) and
            bucket_end >= 
    dateadd(day, cast('-2' as integer), cast(max_bucket_end as timestamp))
 and
            training_set_size >= 7
          then TRUE else FALSE end as is_anomalous
        from anomaly_scores
      )

      select
        metric_value as value,
        training_avg as average,
        
        case
        when is_anomalous = TRUE and 'both' = 'spike' then
         lag(metric_value) over (partition by full_table_name, column_name, metric_name, dimension, dimension_value, bucket_seasonality order by bucket_end)
        when is_anomalous = TRUE and 'both' != 'spike' then
         lag(min_metric_value) over (partition by full_table_name, column_name, metric_name, dimension, dimension_value, bucket_seasonality order by bucket_end)
        when 'both' = 'spike' then metric_value
        else min_metric_value end as min_value,
        case
        when is_anomalous = TRUE and 'both' = 'drop' then
         lag(metric_value) over (partition by full_table_name, column_name, metric_name, dimension, dimension_value, bucket_seasonality order by bucket_end)
        when is_anomalous = TRUE and 'both' != 'drop' then
         lag(max_metric_value) over (partition by full_table_name, column_name, metric_name, dimension, dimension_value, bucket_seasonality order by bucket_end)
        when 'both' = 'drop' then metric_value
        else max_metric_value end as max_value,
        bucket_start as start_time,
        bucket_end as end_time,
        *
      from anomaly_scores_with_is_anomalous
      order by bucket_end, dimension_value
) results
      where
        anomaly_score is not null and
        upper(full_table_name) = upper(cast('CHM_CORE_PROD.BEHAVIORAL.PAGE_VIEWS' as varchar)) and
        metric_name = cast('average' as varchar)
          and upper(column_name) = upper(cast('ENGAGED_TIME_IN_S' as varchar))
thats the query the report UI gives here
image.png
this is the definition of the column_anomaly
n
Are you trying to run the result query in your Snowflake?
d
yup
image.png
Object 'CHM_CORE_PROD.ELEMENTARY."test_bdf5920709_elementary_column_anomalies_page_views_average__ENGAGED_TIME_IN_S__day_of_week__3__day__1__anomaly_scores__tmp_20231116024747364356"' does not exist or not authorized.
n
This is a temp table, which during the run is created (the name ends with
tmp_<timestamp>
).
d
yeah but why does the result query reference it?
btw i saw an earlier issue that sounds similar
n
The result of this test is calculated using a temp table. This is why this table is part of the query (this is the SQL that was run). Keeping these tables alive after the run will use more space in your DWH. If it is something that you find useful, please open a feature request and the team will prioritize it 🙂 💫
❤️ 1
d
ahhh makes sense! thanks for the feedback 🙂
🙌🏼 1