`RowSignature` currently does an order sensitive equality ch Apache Druid #dev

`RowSignature` currently does an order sensitive e...

Atul Mohan

08/14/2025, 10:04 PM

RowSignature

currently does an order sensitive equality check here. Do we actually need to preserve column order or is it sufficient to check if both the objects have all the columns regardless of order? I've observed instances where BrokerSegmentMetadataCache frequently updates the row signature, with the only difference being a change in column order. These frequent updates cause signature changes during the query planning phase, In rare cases, the broker generates the projection based on the old signature and apply it to the new signature leading to unexpected query results.

Atul Mohan

08/15/2025, 7:57 PM

Any thoughts here @Clint Wylie @Rishabh?

Atul Mohan

08/25/2025, 10:35 PM

I've added more info here: https://github.com/apache/druid/issues/18437

Gian Merlino

08/27/2025, 10:49 AM

the order does matter in some cases where

RowSignature

is used (such as in interpreting array-based results)

Gian Merlino

08/27/2025, 10:51 AM

In rare cases, the broker generates the projection based on the old signature and apply it to the new signature leading to unexpected query results.

This is pretty wild and definitely should be fixed… the effective signature of a table should be locked down once query planning starts

Gian Merlino

08/27/2025, 10:58 AM

Independently of that issue (which we definitely should fix) I thought we had some mechanism to stabilize the signatures of tables… IIRC, the logic builds the signature by merging physical segment signatures in a consistent order, so even if pre-existing segments don't all agree on the order, you do get a stable order for SQL unless you add new segments that change the order

Gian Merlino

08/27/2025, 10:58 AM

are you adding new segments that change the order frequently? Maybe using column discovery?

Atul Mohan

08/27/2025, 10:12 PM

@Gian Merlino There are new segments that get created but they may not always have all the columns. For example, if the current datasource signature is:

Copy code

[inline_data] has new signature: {__time:LONG, dim1:STRING, dim3:STRING, d1:STRING, d2:STRING, d3:STRING}.

And I add a new segment with dimensions

dim1

and

d1

It changes the row signature to :

Copy code

[inline_data] has new signature: {__time:LONG, dim1:STRING, d1:STRING, dim3:STRING, d2:STRING, d3:STRING}.

Is this the expected behavior? The new segment doesn't really change the order but it doesn't have all the columns from the signature.

Open in Slack

Previous Next