hi team, why are (our) volume tests ridiculously s...
# community-support
m
hi team, why are (our) volume tests ridiculously slow? we’re on databricks (dbt 1.7, elementary 0.13), where such a query would be instantaneous, irrespective of table size/number of rows/number of partitions (due to partition metadata, assuming
date
being a partition):
Copy code
select date, count(*) from table group by date
why then would a default volume test with
timestamp_column: date
as only argument take many minutes to execute? is that a known (databricks) issue?
ok, now i found why: the first thing volume tests execute is (simplified)
Copy code
select timestamp_column
from table
where timestamp_column >= training_period
and only then calculate the
buckets
and only then calculate the
counts
of the above select per
bucket
. would it not be an option to calculate the
buckets
first, and then query the
table
to directly get the
counts
of
timestamp_column
per
bucket
? this way, if
timestamp_column
is a partition (or clustered etc) and its grain is the same as
time_bucket:period
, you would reduce runtime to almost zero. this might have a very noticable effect on companies running a lot of (standard) volume tests.
shameless ping (again) to @Noa Kurman
this seems to be very related (thx @Francesco Mucio for finding)
fyi @Samuel Garfield
👍 1
👀 1
s
I feel like I'm going crazy because I swear I submitted another feature request as well to pre-aggregate the time buckets in earlier queries so the anomaly tests are more performant as well, but I cannot for the life of me find it
ah I accidentally submitted it as a dicussion instead of an issue. New issue here
This one may also be of interest to you @michael böhm
m
thanks!
f
nice one
I guess the change needs to be applied here
👍 1
or actually write a new macro just for the volumes
m
Noa sneakily suggested to ping @Ella Katz for this blob wave
n
Hey all! Sorry for the delay here Actually this is a known issue, our queries as you mentioned are not optimized to work with partitions Fortunately this is already in our backlog and we’re planning on working on it soon 🙂
🙌 1
garfieldthumbsup 1
m
hi @Noa Kurman, any news on this? (still the most-dearly-missed test for us)
regular 2-months inquiry, @Noa Kurman 😉
@Itamar Hartstein i see you tentatively closed the related issue from Sam, but at least for us the above isn’t resolved - as in the tests still do not make use of existing table/column optimizations and take many minutes instead of few seconds. would the team still like to address this?
g
Hi team, are there any news here? @Noa Kurman @Itamar Hartstein Is there a way we can see where on your backlog this is atm? Similar to Michael, this would not be usable for larger-volume datasets, which are also unfortunately the ones we’d like to test the most for volume 😞
🙇 1
m
seems they won’t answer anything in support, and they won’t react to the github issue because they prematurely closed it. only chance is to open a new issue duplicate and hope it won’t get ignored too.
which is now here