michael böhm
12/12/2023, 3:21 PMdate being a partition):
select date, count(*) from table group by date
why then would a default volume test with timestamp_column: date as only argument take many minutes to execute?
is that a known (databricks) issue?michael böhm
12/13/2023, 10:57 AMselect timestamp_column
from table
where timestamp_column >= training_period
and only then calculate the buckets and only then calculate the counts of the above select per bucket.
would it not be an option to calculate the buckets first, and then query the table to directly get the counts of timestamp_column per bucket?
this way, if timestamp_column is a partition (or clustered etc) and its grain is the same as time_bucket:period, you would reduce runtime to almost zero. this might have a very noticable effect on companies running a lot of (standard) volume tests.michael böhm
12/13/2023, 11:00 AMmichael böhm
12/13/2023, 3:34 PMmichael böhm
12/13/2023, 3:41 PMSamuel Garfield
12/14/2023, 2:29 AMSamuel Garfield
12/14/2023, 2:35 AMSamuel Garfield
12/14/2023, 2:37 AMmichael böhm
12/14/2023, 7:46 AMFrancesco Mucio
12/14/2023, 9:32 AMFrancesco Mucio
12/14/2023, 9:37 AMFrancesco Mucio
12/14/2023, 9:39 AMmichael böhm
01/02/2024, 3:25 PMNoa Kurman
01/10/2024, 12:45 PMmichael böhm
03/11/2024, 9:29 AMmichael böhm
05/16/2024, 1:18 PMmichael böhm
06/07/2024, 2:00 PMGeorgi Evlogiev
08/06/2024, 2:19 PMmichael böhm
08/07/2024, 8:14 AMmichael böhm
08/07/2024, 8:26 AM