Hi, I've recently upgraded from 3.4.0 to 3.4.8 and...
# questions-and-troubleshooting
n
Hi, I've recently upgraded from 3.4.0 to 3.4.8 and found that my partitioned materialized views were negatively affected and I am not sure if the behavior change is intended or a bug. Upgrading turned some partitioned MVs into unpartitioned MVs, and attempting to use the SQL code to create them before fails. Below is a simplified and obfuscated example of one of the related MVs. The two OLAP tables used here are both partitioned exactly the same: from_unixtime(unix_timestamp,'%Y-%m-%d'). The error message is "> 1064 - Getting analyzing error. Detail message: Column 'date_field' cannot be resolved." Attempting to remove the alias and use __generated_partition_column_0 throughout the whole query results in a different message "> 1064 - Unknown error". I believe this is related to using identical generated partition column from multiple tables, since I run into similar problems when attempting aggregations on unions between partitioned tables that use this partition strategy, but it works when using only one table.
Copy code
CREATE MATERIALIZED VIEW mv_partitioned_1
REFRESH ASYNC
PARTITION BY date_field
AS
WITH
cte_1 AS (
	select 
		field1,
		field2,
		__generated_partition_column_0 as date_field	
	FROM 
		table1			
	GROUP BY field1, field2, date_field 
),
cte_2 AS (
	select 
		field1,
		field2,
		__generated_partition_column_0 as date_field
	FROM 
		table2
	GROUP BY field1, field2, date_field 
)
SELECT DISTINCT
	cte_1.field1,
	cte_1.field2
	cte_2.field2,
	cte_1.date_field
FROM 
	cte_1 JOIN
	cte_2 oc ON cte_1.field1 = cte_2.field1
Another issue I believe may be related. This MV kept its partition when upgrading unlike the one above, but refresh time for each partition has increased by a factor of 5, with the only change to the system being the upgrade from 3.4.0 -> 3.4.8. Is it possible that changes to the MV analyzer related to generated partition columns could have an affect here?
Copy code
CREATE MATERIALIZED VIEW mv_partitioned_2
REFRESH ASYNC
PARTITION BY date_field
AS
SELECT 
	field_1, 
	date_field, 
	max(max_field_2) AS max_field_2,
FROM (
	SELECT 
		CAST(json_object->'field_1' as int) as field_1,
		max(CAST(json_object->'field_2') as int) as max_field_2,
		__generated_partition_column_0 as date_field		
	FROM 
		table, 
		UNNEST(json_objects_1) AS t(json_object)		
	GROUP BY CAST(json_object->'field_1' as int), __generated_partition_column_0
	union
	SELECT 
		CAST(json_object->'field_1' as int) as field_1,
		max(CAST(json_object->'field_2') as int) as max_field_2,
		__generated_partition_column_0 as date_field		
	FROM 
		table, 
		UNNEST(json_objects_2) AS t(json_object)		
	GROUP BY CAST(json_object->'field_1' as int), __generated_partition_column_0
	) AS tbl
GROUP BY field_1, date_field
r
For the column name issue, this is most likely a bug - the parser mixed the column name. Can you please submit a github issue and send the ticket here? For the latency, can you share the query profile for the refresh? If the profile is not enabled yet:
set enable_profile = true; --enable the query profile
refresh materialized view mv_partitioned_2;
select * from information_schema.task_runs;--obtain the query id
select get_query_profile("<query-id>");