Hi team, first of all: great plugin you have there, clients love see additional observability for their test. However, with 0.11.2 we face the problem that a lot (roughly 50%) of our test runs are not logged (not even written to the elementary schema in the database). I cannot see a clear pattern, but so for it looks like it only affects elementary anomoly/volume/schema detections, no builtin dbt or dbt-expectations tests. but not all elementary tests are affected, still trying to find out a pattern here. Downgrading to 0.11.1 fixes this problem for us. I am happy to investigate any further if you have any pointers where to look into deeper

Hi @Matthias Niehoff, What DWH are you using?

Just a simple Postgres

we also lost the majority of our tests from the report today. Running on Snowflake I noticed that if I run a test individually with a tag, and then use

edr report.

That test would show up. Tried this with a volume test. However, when I ran a whole lot of tests with the same tag, only some of them showed up on the report. downgrading to edr 0.11.0 and elementary 0.11.1 fixed everything

We just released 0.12.0 (package and python), can you please check if the issue persists there as well?

will do in a few min

0.12.0 on both 🙏🏼

0.12.0 edr and 0.12.0 elementary has all the tests showing. Issue appears to be gone

can’t confirm here. Also with 0.12.0 the majority of elementary test runs is still missing. selecting them implicitly works as damian pointed out

@Matthias Niehoff thanks for flagging the issue, we will look into this. Can you send me your dbt.log containing both runs in a DM?

@Ella Katz after a ton of testing today. I have to redact my initial statement that 0.12.0 fixed the issue. I ran on a clean elementary schema today with 0.12.0 and could clearly see that

dbt test

actually ran all the tests, and the console showed the warnings and errors (around ~1000 tests with about a dozen warnings and a couple of errors). But the elementary.elementary_test_results table only had 133 records (which is what the cli report showed) Tried running all the tests again by giving them the same tag and running

dbt test -s tag:elementary_poc

. Again hundreds of tests ran and their output was visible on console. However, the report only went up to 251 rows. There is no obvious pattern for why some tests are in and some are out. For example, only one freshness test was showing, even though I have 40+ tables set up the same way. Next, I downgraded to edr 0.11.0 and elementary 0.11.1 on an empty elementary schema. The output was long again, but the elementary.elementary_test_results table now has ~1000 records and the report looks much closer to how it used to. Finally, I upgraded to edr 0.11.0 and elementary 0.11.2 and once again emptied the elementary schema. The output was long again, but the elementary.elementary_test_results table now has 135 records and the report looks much closer to how it used to. to recap: edr 0.12.0 - elementary 0.12.0 = 133 tests edr 0.11.0 - elementary 0.11.1 = 1009 tests edr 0.11.0 - elementary 0.11.2 = 135 tests same config on all 3 tests

Hi @Damian Eralio, Thanks for the info. Checking the issue

Hey @Damian Eralio, we released

0.12.1

with a fix. Thank you for reporting!

@Maayan Salom I've updated to 0.12.1 and executed with edr 0.12.0. Everything looks good after a couple of runs. Thank you for your hard work!