[SPARK-58757][SQL] Allow CollapseWindow to merge windows with an empty order spec

[ulysses-you]

What changes were proposed in this pull request?

Currently, CollapseWindow collapses two adjacent Window operators only when their partition specs and order specs are identical. This PR relaxes it to also merge two windows with the same partition spec when one of them has an empty order spec, as long as every window expression of the empty-order window is order-insensitive.

A window expression is treated as order-insensitive when its frame is the whole partition (ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING): such a frame always covers every row of the partition regardless of the ordering, so aggregates like count/sum/min/max give the same value under any ordering, and functions whose result does depend on the row order (e.g. collect_list, first) are non-deterministic when the order spec is empty, so evaluating them under any ordering yields a valid result. Windows with a bounded frame (e.g. ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) are order-sensitive and are never merged.

The merged window keeps the non-empty order spec of the other window.

Why are the changes needed?

For queries that mix an ordered window function with an unordered aggregate over the same partition, e.g.:

SELECT c2, c1,
       row_number() OVER (PARTITION BY c1 ORDER BY c2) AS rk,
       count(1)     OVER (PARTITION BY c1)
FROM t3

the current rule keeps two Window operators even though they share PARTITION BY c1. After this change they are collapsed into a single window operator, saving one WindowExec pass (the [c1, c2] sort is already shared in both plans). A local benchmark on 4M rows showed roughly 16% faster runtime in the non-spill case and 20% in the spill case.

Does this PR introduce any user-facing change?

The query result is unchanged for the common shape (the empty-order window written after the ordered one), where the merge does not change the input order of any window expression. When the empty-order window appears before an ordered sibling in the SELECT list, the merge evaluates its expressions under the sibling's order; for functions documented as non-deterministic without an order (first, last, collect_list) the value may differ, which is already allowed by their contract. FP sum/avg may also differ at the bit level, consistent with Spark's existing treatment of FP aggregation as order-sensitive (EliminateSorts.isOrderIrrelevantAggs).

How was this patch tested?

Added tests to CollapseWindowSuite covering:

• collapse when the empty-order window has a whole-partition frame (row_number + count), including the case where it is the inner window;
• collapse when the empty-order window has multiple window expressions;
• collapse when the empty-order window has first over the whole partition;
• the SPARK-34565 shape with a Project between the windows;
• no collapse when the empty-order window has a bounded frame.

Ran CollapseWindowSuite (13 tests) and TransposeWindowSuite (8 tests), all pass.

Was this patch authored or co-authored using generative AI tooling?

Yes, developed with assistance from Claude Code.

Generated-by: Claude Code

[uros-b]

LGTM, thank you @ulysses-you!