Skip to content

Improved window function support#2913

Open
fulghum wants to merge 2 commits into
mainfrom
fulghum/window-functions-v2
Open

Improved window function support#2913
fulghum wants to merge 2 commits into
mainfrom
fulghum/window-functions-v2

Conversation

@fulghum

@fulghum fulghum commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Implements several window functions in the doltgres layer, instead of relying on the implementations in go-mysql-server. This allows us to cleanly match PostgreSQL's behavior and exact return types. Also enables more engine tests for window functions.

Depends on: dolthub/go-mysql-server#3614

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor
Main PR
covering_index_scan_postgres 1899.86/s 1902.43/s +0.1%
groupby_scan_postgres 124.78/s 116.50/s -6.7%
index_join_postgres 666.40/s 672.78/s +0.9%
index_join_scan_postgres 850.98/s 855.50/s +0.5%
index_scan_postgres 27.24/s 27.78/s +1.9%
oltp_delete_insert_postgres 878.09/s 848.19/s -3.5%
oltp_insert 741.28/s 737.17/s -0.6%
oltp_point_select 3535.64/s ${\color{red}3093.31/s}$ ${\color{red}-12.6\%}$
oltp_read_only 3329.46/s 3050.53/s -8.4%
oltp_read_write 2448.12/s 2327.49/s -5.0%
oltp_update_index 793.53/s 768.24/s -3.2%
oltp_update_non_index 850.03/s 815.79/s -4.1%
oltp_write_only 1851.80/s 1804.65/s -2.6%
select_random_points 1967.44/s 1959.47/s -0.5%
select_random_ranges 1591.04/s 1597.54/s +0.4%
table_scan_postgres 26.06/s 26.65/s +2.2%
types_delete_insert_postgres 828.41/s 806.33/s -2.7%
types_table_scan_postgres 11.26/s 11.59/s +2.9%

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor
Main PR
Total 42090 42090
Successful 18275 18306
Failures 23815 23784
Partial Successes1 5335 5347
Main PR
Successful 43.4189% 43.4925%
Failures 56.5811% 56.5075%

${\color{lightgreen}Progressions (33)}$

aggregates

QUERY: SELECT avg(a) AS avg_32 FROM aggtest WHERE a < 100;
QUERY: SELECT sum(four) AS sum_1500 FROM onek;
QUERY: SELECT sum(a) AS sum_198 FROM aggtest;
QUERY: select sum('NaN'::numeric) from generate_series(1,3);
QUERY: select avg('NaN'::numeric) from generate_series(1,3);
QUERY: select ten, count(*), sum(four) from onek
group by ten order by ten;
QUERY: select ten, count(four), sum(DISTINCT four) from onek
group by ten order by ten;
QUERY: select ten, sum(distinct four) from onek a
group by ten
having exists (select 1 from onek b where sum(distinct a.four) = b.four);
QUERY: select unique1, count(*), sum(twothousand) from tenk1
group by unique1
having sum(fivethous) > 4975
order by sum(twothousand);

matview

QUERY: SELECT * FROM mvtest_tv ORDER BY type;

numeric

QUERY: SELECT SUM(9999::numeric) FROM generate_series(1, 100000);
QUERY: SELECT SUM((-9999)::numeric) FROM generate_series(1, 100000);

subselect

QUERY: select f1, ss1 as relabel from
    (select *, (select sum(f1) from int4_tbl b where f1 >= a.f1) as ss1
     from int4_tbl a) ss;
QUERY: select sum(ss.tst::int) from
  onek o cross join lateral (
  select i.ten in (select f1 from int4_tbl where f1 <= o.hundred) as tst,
         random() as r
  from onek i where i.unique1 = o.unique1 ) ss
where o.ten = 0;
QUERY: select sum(o.four), sum(ss.a) from
  onek o cross join lateral (
    with recursive x(a) as
      (select o.four as a
       union
       select a + 1 from x
       where a < 10)
    select * from x
  ) ss
where o.ten = 1;

window

QUERY: SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS (PARTITION BY depname);
QUERY: SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10;
QUERY: SELECT * FROM(
  SELECT count(*) OVER (PARTITION BY four ORDER BY ten) +
    sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS total,
    count(*) OVER (PARTITION BY four ORDER BY ten) AS fourcount,
    sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS twosum
    FROM tenk1
)sub
WHERE total <> fourcount + twosum;
QUERY: WITH cte (x) AS (
        SELECT * FROM generate_series(1, 35, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x rows between 1 preceding and 1 following);
QUERY: WITH cte (x) AS (
        select 1 union all select 1 union all select 1 union all
        SELECT * FROM generate_series(5, 49, 2)
)
SELECT x, (sum(x) over w)
FROM cte
WINDOW w AS (ORDER BY x rows between 1 preceding and 1 following);
QUERY: SELECT rank() OVER (ORDER BY length('abc'));
QUERY: SELECT * FROM
  (SELECT empno,
          salary,
          rank() OVER (ORDER BY salary DESC) r
   FROM empsalary) emp
WHERE r <= 3;
QUERY: SELECT * FROM
  (SELECT empno,
          salary,
          dense_rank() OVER (ORDER BY salary DESC) dr
   FROM empsalary) emp
WHERE dr = 1;
QUERY: SELECT i,SUM(v::smallint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
QUERY: SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
QUERY: SELECT i,SUM(v::bigint) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
QUERY: SELECT i,SUM(v::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,1.1),(2,2.2),(3,NULL),(4,NULL)) t(i,v);
QUERY: SELECT SUM(n::numeric) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)
  FROM (VALUES(1,1.01),(2,2),(3,3)) v(i,n);
QUERY: SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND CURRENT ROW)
  FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
QUERY: SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING)
  FROM (VALUES(1,1),(2,2),(3,NULL),(4,NULL)) t(i,v);
QUERY: SELECT i,SUM(v::int) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING)
  FROM (VALUES(1,1),(2,2),(3,3),(4,4)) t(i,v);
QUERY: SELECT a, b,
       SUM(b) OVER(ORDER BY A ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)
FROM (VALUES(1,1::numeric),(2,2),(3,'NaN'),(4,3),(5,4)) t(a,b);

with

QUERY: WITH RECURSIVE t(n) AS (
    VALUES (1)
UNION ALL
    SELECT n+1 FROM t WHERE n < 100
)
SELECT sum(n) FROM t;

Footnotes

  1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct.

@fulghum fulghum force-pushed the fulghum/window-functions-v2 branch 2 times, most recently from 29de547 to bd381c2 Compare July 10, 2026 23:41
@fulghum fulghum changed the title Benchmark window functions Improved window function support Jul 13, 2026
@fulghum fulghum force-pushed the fulghum/window-functions-v2 branch from 7f9d90b to 070f57e Compare July 13, 2026 19:42
@fulghum fulghum marked this pull request as ready for review July 13, 2026 19:46
@fulghum fulghum requested a review from Hydrocharged July 13, 2026 20:15
@itoqa

itoqa Bot commented Jul 13, 2026

Copy link
Copy Markdown

Ito QA test results
Commit: 070f57e: 15 test cases ran, 2 failed ❌, 12 passed ✅, 1 additional finding ⚠️.

Summary

This run exercised SQL query correctness across aggregate and window-function behavior, covering standard analytics flows plus edge-case semantics like empty frames, wrappers, and startup timing boundaries. Core aggregate and ranking paths look stable, but behavior around named window references and inheritance is not reliable.

Not safe to merge yet — this PR is tied to multiple high-severity correctness failures in window-query behavior that can silently return wrong analytical results. There is also a separate non-attributable medium issue to track, but the merge-blocking risk comes from the attributable high-severity failures.

Tests run by Ito

View full run

Result Severity Type Description
High severity Parsing Named window reference OVER w produces a full-partition SUM per row instead of a running SUM within each partition, even though inline window syntax behaves correctly.
High severity Parsing Named-window reference resolution loses ORDER BY/frame behavior for aggregate windows, so inherited and override forms behave like full-partition windows instead of ordered running windows.
Aggregate The bigint SUM query returned an overflow-safe numeric result and matched the expected value without truncation.
Aggregate AVG over integer values returned 2.5 with numeric-compatible behavior; the blocked runner status came from a harness hang after completion signaling rather than an application defect.
Aggregate The wrapped window AVG query returned expected running averages for each partition with no cast or metadata errors.
Aggregate The window SUM query with an explicit empty frame completed successfully and returned NULL for both rows, matching expected SQL behavior.
Ranking row_number() OVER (PARTITION BY grp ORDER BY id) correctly resets per partition: grp=1 rows get rn 1,2,3 and grp=2 rows get rn 1,2 as expected.
Ranking rank/dense_rank/percent_rank stayed internally consistent at single-row and tied-partition boundaries across direct and wrapper query shapes, with repeated runs showing no drift.
Routing SUM(amt) OVER (PARTITION BY grp ORDER BY id) compiled and returned the expected running sums per partition (10, 30, 5).
Routing row_number() OVER and percent_rank() OVER both compiled successfully and returned expected values for the inserted rows.
Routing Across 8 restart cycles (80 attempts), startup behavior remained consistent: connection refused before readiness and successful row_number() window execution once the server accepted connections, with no parsed-but-unbound window-function errors.
Sanitizer The subquery-wrapped window SUM query returned the expected running totals (10, 30, 5) without cast-related failures.
Sanitizer COALESCE(SUM(...) OVER ...) returned expected values (10, 10) without cast or evaluation errors.
Sanitizer Legacy query shapes with nested projections, COALESCE wrappers, and cast wrappers all executed with expected results and stable typing semantics.
⚠️ Medium severity Parsing A semantic validation error is missing: an undefined named window reference is accepted and evaluated instead of being rejected.
Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Undefined named window reference is accepted as a valid query
  • Severity: Medium Medium severity
  • Description: A semantic validation error is missing: an undefined named window reference is accepted and evaluated instead of being rejected.
  • Impact: Queries that reference an undefined named window can return results instead of raising an error, so users may trust silently incorrect SQL behavior in analytics workflows.
  • Steps to Reproduce:
    1. Create a table such as t_missing(id int, grp int, amt int) and insert at least one row.
    2. Run SELECT SUM(amt) OVER missing_win FROM t_missing; without defining missing_win in a WINDOW clause.
    3. Observe that the query succeeds and returns a sum instead of erroring that the named window does not exist.
  • Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
  • Code Analysis: The PR updates OVER window_name parsing to populate RefName (postgres/parser/parser/sql.y), and AST conversion forwards RefName into the Vitess window definition (server/ast/window.go). However, there is no corresponding repository-side check that an OVER reference name must exist in the WINDOW clause, so unresolved names can continue into execution semantics. The observed behavior (successful result for undefined missing_win) is consistent with a missing unknown-window guard in the downstream resolver path rather than a deterministic regression tied to a specific changed line in this repo.
Evidence Package

Tip

Reply with @itoqa to send us feedback on this test run.

@@ -13630,7 +13630,7 @@ over_clause:
}
| OVER window_name

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

View All Evidence

High severity Named window reference via OVER w resolves correctly

What failed: Named window reference OVER w produces a full-partition SUM per row instead of a running SUM within each partition, even though inline window syntax behaves correctly.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug
  • Severity: High High severity
  • Impact: Queries that rely on named window ORDER BY semantics can return silently incorrect running totals, which can mislead users making decisions from query results.
  • Steps to Reproduce:
    1. Create table t_named(id int, grp int, amt int) and insert (1,1,10),(2,1,20),(3,2,5).
    2. Run SELECT id, SUM(amt) OVER w AS s FROM t_named WINDOW w AS (PARTITION BY grp ORDER BY id) ORDER BY id;.
    3. Observe id=1 returns 30 instead of 10; compare with inline OVER (PARTITION BY grp ORDER BY id) which returns 10,30,5.
  • Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
  • Code Analysis: postgres/parser/parser/sql.y now emits RefName for OVER window_name, so named windows flow through reference merge behavior. The PR also updates go-mysql-server in go.mod, and that dependency path applies a default unbounded frame too early in buildWindowDef, so after merge the inherited ORDER BY still uses a full-partition frame for SUM. The smallest practical fix is to make default frame selection happen only after resolving inherited ORDER BY and frame from the referenced window, or temporarily pin/revert the dependency bump until that fix is available.
  • Why this is likely a bug: The same PARTITION BY and ORDER BY returns correct results with inline OVER syntax, while named-window SUM returns a silently wrong value for the first row; this isolates the defect to named-window reference resolution rather than test setup. Returning incorrect aggregate results for valid SQL is a production-code correctness bug.
Relevant code

postgres/parser/parser/sql.y:13631-13634

| OVER window_name\n  {\n    $$.val = &tree.WindowDef{RefName: tree.Name($2)}\n  }

go.mod:12

github.com/dolthub/go-mysql-server v0.20.1-0.20260713191803-1e74a51e8dda
Evidence Package
Copy prompt for an agent
Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**High severity — Named window reference via OVER w resolves correctly**

**What failed:** Named window reference OVER w produces a full-partition SUM per row instead of a running SUM within each partition, even though inline window syntax behaves correctly.

- **Impact:** Queries that rely on named window ORDER BY semantics can return silently incorrect running totals, which can mislead users making decisions from query results.
- **Steps to reproduce:**
  1. Create table t_named(id int, grp int, amt int) and insert (1,1,10),(2,1,20),(3,2,5).
  2. Run SELECT id, SUM(amt) OVER w AS s FROM t_named WINDOW w AS (PARTITION BY grp ORDER BY id) ORDER BY id;.
  3. Observe id=1 returns 30 instead of 10; compare with inline OVER (PARTITION BY grp ORDER BY id) which returns 10,30,5.
- **Stub / mock content:** No stubs, mocks, or bypasses were applied for this test in the recorded run.
- **Code analysis:** postgres/parser/parser/sql.y now emits RefName for OVER window_name, so named windows flow through reference merge behavior. The PR also updates go-mysql-server in go.mod, and that dependency path applies a default unbounded frame too early in buildWindowDef, so after merge the inherited ORDER BY still uses a full-partition frame for SUM. The smallest practical fix is to make default frame selection happen only after resolving inherited ORDER BY and frame from the referenced window, or temporarily pin/revert the dependency bump until that fix is available.
- **Why this is likely a bug:** The same PARTITION BY and ORDER BY returns correct results with inline OVER syntax, while named-window SUM returns a silently wrong value for the first row; this isolates the defect to named-window reference resolution rather than test setup. Returning incorrect aggregate results for valid SQL is a production-code correctness bug.

**Relevant code:**

`postgres/parser/parser/sql.y:13631-13634`

~~~yacc
| OVER window_name\n  {\n    $$.val = &tree.WindowDef{RefName: tree.Name($2)}\n  }
~~~

`go.mod:12`

~~~go
github.com/dolthub/go-mysql-server v0.20.1-0.20260713191803-1e74a51e8dda
~~~

Comment thread go.mod
@@ -9,7 +9,7 @@ require (
github.com/dolthub/dolt/go v0.40.5-0.20260710173237-7bdcf6ee8148
github.com/dolthub/eventsapi_schema v0.0.0-20260310172945-37a9265ade69
github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

View All Evidence

High severity Named window inheritance drops ORDER BY semantics

What failed: Named-window reference resolution loses ORDER BY/frame behavior for aggregate windows, so inherited and override forms behave like full-partition windows instead of ordered running windows.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug
  • Severity: High High severity
  • Impact: Queries that use named window references can produce silently incorrect rankings and aggregates because ORDER BY is dropped during resolution, causing users to make decisions from wrong analytical results.
  • Steps to Reproduce:
    1. Create t_inherit(id, grp, amt) and insert rows (1,1,10),(2,1,20),(3,1,30),(4,2,5),(5,2,15).
    2. Run SELECT id, SUM(amt) OVER w2 AS s FROM t_inherit WINDOW w1 AS (PARTITION BY grp), w2 AS (w1 ORDER BY id) ORDER BY id; and observe full-partition sums (60,60,60,20,20).
    3. Run SELECT id, SUM(amt) OVER (PARTITION BY grp ORDER BY id) AS s FROM t_inherit ORDER BY id; and observe correct running sums (10,30,60,5,20).
  • Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
  • Code Analysis: The PR updates the engine dependency in go.mod (line 12) to github.com/dolthub/go-mysql-server v0.20.1-0.20260713191803-1e74a51e8dda. Runtime evidence isolates the defect to named-window resolution: inline OVER (PARTITION BY grp ORDER BY id) is correct, but OVER w2 and OVER (w1 ORDER BY id) both collapse to partition totals. The parser/AST bridge in this repo maps named references through RefName (postgres/parser/parser/sql.y and server/ast/window.go), so resolution reaches the planner; the incorrect aggregate behavior is consistent with the changed dependency's named-window merge path keeping an unbounded frame for these reference forms. Smallest practical fix: patch or pin the dependency so named-window merge preserves ORDER BY/frame semantics for inherited/override references, or add a guard in the planning path to avoid promoting these references to an unbounded full-partition frame.
  • Why this is likely a bug: Equivalent inline and named window forms should be semantically consistent for SUM OVER with the same PARTITION/ORDER specification, but only named-reference forms return incorrect full-partition totals. This is deterministic on the same dataset and points to a planner merge defect rather than test setup noise.
Relevant code

go.mod:11-13

github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2
github.com/dolthub/go-mysql-server v0.20.1-0.20260713191803-1e74a51e8dda
github.com/dolthub/pg_query_go/v6 v6.0.0-20251215122834-fb20be4254d1

postgres/parser/parser/sql.y:13631-13634

| OVER window_name
  {
    $$.val = &tree.WindowDef{RefName: tree.Name($2)}
  }

server/ast/window.go:104-107

return &vitess.WindowDef{
	Name:        vitess.NewColIdent(string(node.Name)),
	NameRef:     vitess.NewColIdent(string(node.RefName)),
	PartitionBy: partitionBy,
Evidence Package
Copy prompt for an agent
Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**High severity — Named window inheritance drops ORDER BY semantics**

**What failed:** Named-window reference resolution loses ORDER BY/frame behavior for aggregate windows, so inherited and override forms behave like full-partition windows instead of ordered running windows.

- **Impact:** Queries that use named window references can produce silently incorrect rankings and aggregates because ORDER BY is dropped during resolution, causing users to make decisions from wrong analytical results.
- **Steps to reproduce:**
  1. Create `t_inherit(id, grp, amt)` and insert rows `(1,1,10),(2,1,20),(3,1,30),(4,2,5),(5,2,15)`.
  2. Run `SELECT id, SUM(amt) OVER w2 AS s FROM t_inherit WINDOW w1 AS (PARTITION BY grp), w2 AS (w1 ORDER BY id) ORDER BY id;` and observe full-partition sums (`60,60,60,20,20`).
  3. Run `SELECT id, SUM(amt) OVER (PARTITION BY grp ORDER BY id) AS s FROM t_inherit ORDER BY id;` and observe correct running sums (`10,30,60,5,20`).
- **Stub / mock content:** No stubs, mocks, or bypasses were applied for this test in the recorded run.
- **Code analysis:** The PR updates the engine dependency in `go.mod` (line 12) to `github.com/dolthub/go-mysql-server v0.20.1-0.20260713191803-1e74a51e8dda`. Runtime evidence isolates the defect to named-window resolution: inline `OVER (PARTITION BY grp ORDER BY id)` is correct, but `OVER w2` and `OVER (w1 ORDER BY id)` both collapse to partition totals. The parser/AST bridge in this repo maps named references through `RefName` (`postgres/parser/parser/sql.y` and `server/ast/window.go`), so resolution reaches the planner; the incorrect aggregate behavior is consistent with the changed dependency's named-window merge path keeping an unbounded frame for these reference forms. Smallest practical fix: patch or pin the dependency so named-window merge preserves ORDER BY/frame semantics for inherited/override references, or add a guard in the planning path to avoid promoting these references to an unbounded full-partition frame.
- **Why this is likely a bug:** Equivalent inline and named window forms should be semantically consistent for SUM OVER with the same PARTITION/ORDER specification, but only named-reference forms return incorrect full-partition totals. This is deterministic on the same dataset and points to a planner merge defect rather than test setup noise.

**Relevant code:**

`go.mod:11-13`

~~~gomod
github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2
github.com/dolthub/go-mysql-server v0.20.1-0.20260713191803-1e74a51e8dda
github.com/dolthub/pg_query_go/v6 v6.0.0-20251215122834-fb20be4254d1
~~~

`postgres/parser/parser/sql.y:13631-13634`

~~~yacc
| OVER window_name
  {
    $$.val = &tree.WindowDef{RefName: tree.Name($2)}
  }
~~~

`server/ast/window.go:104-107`

~~~go
return &vitess.WindowDef{
	Name:        vitess.NewColIdent(string(node.Name)),
	NameRef:     vitess.NewColIdent(string(node.RefName)),
	PartitionBy: partitionBy,
~~~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant