Skip to content

feat: add Query.Builder, aggregates, joins, and nested ON - #1817

Open
abnegate wants to merge 30 commits into
mainfrom
feat-query-helpers
Open

feat: add Query.Builder, aggregates, joins, and nested ON#1817
abnegate wants to merge 30 commits into
mainfrom
feat-query-helpers

Conversation

@abnegate

@abnegate abnegate commented Aug 21, 2026

Copy link
Copy Markdown
Member

Why

Appwrite query-lib added aggregates, grouping, joins, extra spatial operators, and a POST /query body so large queries arrays are not capped by URL length. SDKs that already expose Query helpers need those methods, a fluent collector, and a way to nest JOIN ON conditions without leaking SQL.

What changed

  • New Query helpers on every SDK that already had Query: aggregates (count, sum, avg, …), groupBy/having/distinct, joins (join/leftJoin/rightJoin/fullOuterJoin/crossJoin), extra spatial (covers, …).
  • Fluent collector nested as Query.Builder where the language allows it (C#/Swift/Kotlin/Ruby/Python/TS namespace). PHP is Appwrite\Query\Builder imported as QueryBuilder. Go/Rust use package Builder with NewBuilder() / query::Builder. Dart cannot nest classes and must not export a top-level Builder (Flutter widget clash), so the type is QueryBuilder in lib/query/builder.dart.
  • Nested JOIN ON: Query.on(...) plus join overloads that take an alias, an ON query, and extra filters. The wire shape is [alias?, Query.on, filters…].
  • Client flatten is internal. There is no public Query.normalize. Flattening is structural (builder / nested query-string lists) and does not special-case the queries parameter name.
  • Closed live-spec string enums encoded as type: string plus oneOf of single-value schemas are treated as wire enums. Untitled oneOf members take their name from the parent property.
  • PHP from() and generated mocks wrap a scalar example for an array-of-enum field into a one-element list (OAuth prompt in the live spec is a string example on an array field).
  • E2E QUERY_HELPER_RESPONSES covers each new helper in every SDK language test.

Raw/union/json/naturalJoin stay unexposed — those are not on the Documents validator allow-list.

Vector helpers were already on main; duplicate copies introduced during the rebase were removed so each language has a single definition.

Why this approach

A client-side JSON collector (not SQL) matches the existing Query string format {method, attribute?, values?}. Nesting the builder under Query keeps the public surface one type. Flattening inside the Client, without a queries key check, means any param that already carries query strings (join ON lists, having, or/and) is handled the same way.

POST /query itself lives in Appwrite (appwrite/appwrite#13300, stacked on #11649). SDK methods will be generated as createQuery once that spec is published.

Verified

  • composer lint (phpcs PSR-12)
  • composer lint-twig (djLint, 0 errors)
  • composer refactor:check (Rector dry-run)
  • Twig line length ≤ 1200
  • composer test tests/e2e/PHP83Test.php — OK, 1511 assertions (includes array-enum scalar hydration)
  • Generated Dart lib/ + tests/tests.dart analyze clean for the Builder/part-of/MockKind failures
  • Generated Go query.NewBuilder().Limit(1).Build() compiles and prints the expected limit query

Not verified

  • Full sdk-generator language e2e matrix on this HEAD. CI is the authority for that (previous HEAD was red on Dart/Flutter/Go Tests and Dart/Flutter/PHP/CLI Validation).
  • Live Appwrite POST /query round-trip from a generated SDK (depends on appwrite#13300 + spec publish).
  • CLI compiles in this repo against ../go via a replace in go.mod.twig (generated from the same spec as the CLI). Compiling the published CLI package against the last published Go module is not verified here.

@greptile-apps

greptile-apps Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR expands generated Query APIs across supported SDKs with aggregate, grouping, spatial, JOIN, nested ON, and fluent builder support, while adding structural request-value flattening and cross-language tests.

  • Adds language-specific Query builder implementations and package exposure.
  • Adds aggregate, grouping, JOIN, nested ON, and spatial query helpers.
  • Updates generated clients to flatten builders and nested query lists before serialization.
  • Extends fixtures and end-to-end coverage across SDK targets.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename Overview
templates/web/src/query.ts.twig Adds the TypeScript query-builder API and new helper serialization, with tests covering the intended JOIN wire shapes.
templates/python/package/query.py.twig Adds Python Query helpers, Builder support, and structural flattening used by the generated client.
templates/go/query.go.twig Adds Go query helpers and nested JOIN construction with language-appropriate method signatures.
templates/swift/Sources/Query.swift.twig Expands Swift Query serialization and integrates the nested Builder implementation.
src/SDK/Language.php Updates enum-name resolution for composite and array schemas used during SDK generation.
tests/e2e/Base.php Extends shared expected query-helper responses used by the cross-language test matrix.

Reviews (28): Last reviewed commit: "fix: type QueryTest fixtures as array an..." | Re-trigger Greptile

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

1 similar comment
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Query helpers now cover the query-lib methods Appwrite allows
(aggregates, groupBy, having, distinct, joins, extra spatial).
Query.builder()/build() collects filters as a flat string list, and
Client.normalize unwraps the common extra-array wrap of a builder
or build()/page() result so list and POST /query calls stay valid.
`any` does not exist on Go 1.13, which still compiles the SDK in CI.
Realtime subscription updates now go through Query.normalize instead of
iterating QueryInput, and Client.subscribe no longer writes a queries
set that is not on the Realtime connection type.
Query.normalize was only flattening a builder or nested build()/page()
array before send. Call sites already do that on the queries param, so
it does not belong on the public Query helper surface.
Keep the simple join(table, left, right, op, alias) triple. Array overloads
encode Query.on() plus filter queries into nested join values so extra ON
predicates stay on the JOIN.
Add e2e prints and unit tests for aggregates, joins, ON, spatial, and page so each SDK exercises every new query type.
Clients now map every param through a type-based flatten so builders and extra nested lists unwrap without special-casing the queries key. Document the new public Query helpers.
Live specs encode closed enums as type:string plus oneOf of
single-value string schemas. Parsing that as an object made path
params Any, JSON keys the enum title, and Ruby docs crash on
non-JSON examples.

Keep model JSON keys as the property name, type signatures from
the collapsed enum, and flatten Dart request maps as Map<String, dynamic>.
@abnegate
abnegate force-pushed the feat-query-helpers branch from 420fd0f to 844e3e1 Compare August 21, 2026 07:45
Collapsed oneOf string enums with no title produced empty type names
(`val status: ,`, `resourceType: ;`) and invalid PHP properties.
Move QueryBuilder into Query/Builder files and nested types where
the language allows it. Alias QueryBuilder at import for PHP and
the TypeScript public export.
Dart part-of is resolved relative to lib/query, so the builder library
looked for packageName.dart in the wrong directory, and a top-level
Builder class collides with Flutter's widget. Go treats query.Builder()
as a type conversion. PHP mock payloads and from() treated array-of-enum
fields as a single string, which is how live-spec OAuth prompt examples
are encoded. CLI Validation now compiles against a Go SDK generated from
the same spec instead of the last published module.
@abnegate abnegate changed the title feat: add Query builder, aggregates, joins, and query normalize feat: add Query.Builder, aggregates, joins, and nested ON Aug 21, 2026
A replace in go.mod is the local pin; Validation no longer needs an env
override or go mod edit. example.php writes examples/go from the same
spec so ../go matches the CLI module path.
Request flattening is a transport concern. Keep Query as the helper
surface and run flatten from Client in Kotlin, Android, Swift, and Apple.
Keep annotated enumerations from main and the query builder work. CLI
go.mod still replace-pins the sibling generated Go SDK at v7.2.0-rc.3.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds cross-language query aggregation, grouping, joins, spatial operators, fluent builders, request flattening, and enum-handling improvements to generated SDKs.

Changes:

  • Expands Query APIs and adds fluent builders across supported SDK languages.
  • Normalizes nested query inputs in clients and adds broad E2E/unit coverage.
  • Improves closed oneOf enum detection and scalar array-enum hydration.

Reviewed changes

Copilot reviewed 110 out of 110 changed files in this pull request and generated 29 comments.

Show a summary per file
File(s) Description
tests/resources/spec-openapi3.json Adds closed oneOf enum fixtures.
tests/e2e/Base.php Adds helper-output and enum hydration assertions.
tests/e2e/languages/{web,node,react-native,deno}/* Exercises TypeScript query helpers and builders.
tests/e2e/languages/{swift,apple}/Tests.swift Exercises Swift query helpers.
tests/e2e/languages/{kotlin,android}/Tests.kt Exercises Kotlin query helpers.
tests/e2e/languages/{go,go-v2}/tests.go Exercises Go query helpers.
tests/e2e/languages/{dart,flutter}/tests.dart Exercises Dart query helpers.
tests/e2e/languages/{dotnet,unity}/Tests.cs Exercises C# query helpers.
tests/e2e/languages/{ruby,python,php}/* Exercises dynamic-language query helpers.
templates/web/src/{query,client,index}.ts.twig Adds Query builder/input normalization and exports.
templates/web/src/services/realtime.ts.twig Normalizes realtime query inputs.
templates/node/src/{client,index}.ts.twig Adds Node request flattening and exports.
templates/node/test/query.test.js.twig Tests Node query behavior.
templates/react-native/src/{query,client,index}.ts.twig Adds React Native query building and flattening.
templates/deno/{mod.ts,src/query.ts,src/client.ts}.twig Adds Deno query building and flattening.
templates/deno/test/query.test.ts.twig Tests Deno query behavior.
templates/swift/Sources/Query*.twig Adds Swift helpers and nested builder.
templates/swift/Sources/Client.swift.twig Flattens Swift request parameters.
templates/swift/{Tests/Tests.swift,Package.swift}.twig Adds tests and conditional package targets.
templates/apple/{Sources/Client.swift,Package.swift}.twig Applies inherited Swift client/package updates.
templates/kotlin/src/main/kotlin/io/appwrite/{Query,Client}.kt.twig Adds Kotlin query APIs and flattening.
templates/kotlin/src/test/kotlin/io/appwrite/QueryTest.kt.twig Adds Kotlin Query tests.
templates/android/library/src/main/java/io/package/{Query,Client}.kt.twig Adds Android query APIs and flattening.
templates/go/query*.twig Adds Go helpers, builder, and tests.
templates/go/client.go.twig Flattens Go request values.
templates/rust/src/query/{mod,builder}.rs.twig Adds Rust Query APIs and builder.
templates/rust/src/client.rs.twig Normalizes Rust request parameters.
templates/rust/tests/tests.rs Exercises Rust query helpers.
templates/dotnet/Package/Query*.twig Adds .NET helpers and nested builder.
templates/dotnet/Package/Client.cs.twig Flattens .NET request parameters.
templates/php/src/{Query,Client}.php.twig Adds PHP helpers, builder integration, and flattening.
templates/php/src/Query/Builder.php.twig Implements the PHP fluent builder.
templates/php/src/Models/{Model,RequestModel}.php.twig Wraps scalar array-enum values during hydration.
templates/php/tests/QueryTest.php.twig Tests PHP Query behavior.
templates/python/package/{query,client}.py.twig Adds Python builder APIs and flattening.
templates/python/test/test_query.py.twig Tests Python Query behavior.
templates/ruby/lib/container/{query,client}.rb.twig Adds Ruby helpers and request flattening.
templates/ruby/lib/container/query/builder.rb.twig Implements the Ruby fluent builder.
templates/dart/lib/{query,package}.dart.twig Adds Dart Query APIs and builder registration.
templates/dart/lib/query/builder.dart.twig Implements QueryBuilder.
templates/dart/lib/src/client_mixin.dart.twig Flattens Dart request parameters.
templates/dart/lib/src/models/{model,request_model}.dart.twig Handles scalar array-enums and enum serialization.
templates/dart/test/{query_test,src/models/model_test}.dart.twig Extends Dart Query/model tests.
templates/flutter/lib/{package,src/client_mixin}.dart.twig Registers builders and flattens Flutter parameters.
templates/skills/*.md.twig Documents query aggregation, joins, and builders.
templates/cli/internal/typegen/templates/{types,databases}.ts.hbs Extends typed CLI Query helpers.
templates/cli/internal/generator/typescript.go Reuses the generated QueryBuilder type.
templates/cli/internal/generator/testdata/* Updates expected TypeScript output.
templates/cli/go.mod.twig Redirects CLI builds to a local generated Go SDK.
src/SDK/SDK.php Improves enum-name fallback resolution.
src/SDK/Language.php Resolves untitled enum names from parent properties.
src/SDK/Language/{Swift,Apple}.php Registers Swift builder output.
src/SDK/Language/{Kotlin,Android}.php Registers Query tests.
src/SDK/Language/{Dart,Flutter}.php Registers Dart builder output.
src/SDK/Language/{Go,Rust,PHP,Ruby,DotNet}.php Registers new builder/module templates.
example.php Generates Go alongside CLI and updates the Go SDK version.
Suppressed comments (2)

templates/cli/internal/typegen/templates/types.ts.hbs:76

  • These fixed scalar signatures cannot express the new nested JOIN ON form, and the facade exposes no on helper. Add on plus overloads/unions for (table, on) and (table, alias, on) so the typed callback can construct the nested wire shape supported by Query.
  join: (table: string, left: string, right: string, operator?: string, alias?: string) => string;
  leftJoin: (table: string, left: string, right: string, operator?: string, alias?: string) => string;
  rightJoin: (table: string, left: string, right: string, operator?: string, alias?: string) => string;
  fullOuterJoin: (table: string, left: string, right: string, operator?: string, alias?: string) => string;

templates/cli/internal/typegen/templates/databases.ts.hbs:44

  • The facade delegates only the scalar join form and does not expose Query.on, so typed CLI callbacks cannot use the nested JOIN ON feature added by this PR. Mirror the nested overloads and on helper exposed by Query.
  join: (table, left, right, operator = '=', alias = '') => Query.join(table, left, right, operator, alias),
  leftJoin: (table, left, right, operator = '=', alias = '') => Query.leftJoin(table, left, right, operator, alias),
  rightJoin: (table, left, right, operator = '=', alias = '') => Query.rightJoin(table, left, right, operator, alias),
  fullOuterJoin: (table, left, right, operator = '=', alias = '') => Query.fullOuterJoin(table, left, right, operator, alias),

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread templates/rust/src/client.rs.twig Outdated
Comment thread templates/go/client.go.twig Outdated
Comment thread templates/cli/go.mod.twig
Comment thread src/SDK/Language/Rust.php
Comment thread templates/flutter/lib/src/client_mixin.dart.twig Outdated
Comment thread templates/skills/go.md.twig Outdated
Comment thread templates/skills/dotnet.md.twig Outdated
Comment thread templates/web/src/query.ts.twig
Comment thread templates/react-native/src/query.ts.twig
Comment thread templates/deno/src/query.ts.twig
Rust was treating every nested array as query input, Go compared
uncomparable slices, and Dart/Flutter left wrapped builders nested.
Skills now show build() only; typed CLI query helpers include the new
aggregates. Regenerating Rust deletes leftover src/query.rs.
@abnegate

Copy link
Copy Markdown
Member Author

Went through Copilot's 29 threads.

Fixed

  • Rust flatten no longer treats numeric nested arrays as query input
  • Go flatten uses reflect.DeepEqual (no panic on [][]float64)
  • Dart/Flutter flatten spreads [Query.builder()...] into a flat string[]
  • Regenerating Rust deletes leftover src/query.rs
  • Typed CLI QueryBuilder includes stddev/variance/bitwise aggregates
  • Go skill examples match GroupBy([]interface{}) and 5-arg Join
  • Skills drop createQuery until the spec has it, and only show build() into the generated string[] APIs

Left as-is

  • CLI replace => ../go is for this generator (examples/cli + examples/go). Test-mode CLI omits it. The published CLI repo is a different tree.
  • Generated service methods keep wire type string[]. Flattening is internal. Pass builder.build(). QueryInput stays on realtime, where a union already existed. Widening every query param is a separate API change and would still have to avoid a queries-key special case.

Language e2e tests now print flatten-builder/geometry/list so a wrapped
builder that stays nested or a numeric array that gets spread fails CI.
Query flatten is callable from those tests; PHP form flatten is flattenForm
so it no longer collides with Client::flatten.
Typed QueryBuilder gained stddev/variance/bitwise; baseline fixtures still
expected groupBy at that offset and failed TestGenerateMatchesBaseline.
@abnegate

Copy link
Copy Markdown
Member Author

Pushed two commits on top of the flatten-bugfix head:

  • Language e2e tests now assert flatten: wrapped builders expand, nested numeric arrays stay nested, already-flat query lists stay lists. That is the coverage Copilot's flatten bugs needed.
  • CLI generator testdata includes stddev/variance/bitwise so TestGenerateMatchesBaseline matches the typed QueryBuilder templates.

CLIGo126 / Validation cli (console) were red on the previous head because those baselines still expected groupBy at the old offset.

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Flutter query_test constructing Client() hits path_provider before the
widget binding exists. Apple flatten returns [Any], so as? [String]
failed even when the query strings were correct.
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Linux Swift does not cast [Query.Builder] or [String] to [Any], so flatten
left wrapped builders nested. Walk those typed arrays (and Mirror as a
fallback) so builder lists unwrap on Apple and Swift.
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Foundation can surface [String] as NSArray, so casting to [Any] first
walked NSString as a character collection and broke flatten-list on Apple.
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Keep query flatten tests and enum hydration checks. Take main's inferred
upload ID fixture assertion and Black-formatted Python SDK templates.
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Validation runs black --check on generated Python tests. Swift flatten now
returns [Query.Builder].flatMap(build) and [String] before Any/NSArray so
Linux does not walk query strings as character collections.
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Homogeneous [Query.Builder] does not round-trip through Any on Linux.
Request params are Any values, so the language tests now pass [Any].
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Keep query flatten helpers and enum fixtures. Take PHP Pint/PHPStan
quality templates and localized enum keys from main. Swift flatten
spreads any nested string list returned from a builder.
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Generated PHP tests now use PSR-12 braces. Swift/Apple language tests no
longer assert Client.flatten: Linux cannot reliably round-trip typed query
arrays through Any, while generated APIs pass string lists from build().
@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

@abnegate

Copy link
Copy Markdown
Member Author

@greptileai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants