Skip to content

feat(connectors): add SurrealDB sink connector#3453

Open
countradooku wants to merge 25 commits into
apache:masterfrom
countradooku:feat/surrealdb-sink-connector
Open

feat(connectors): add SurrealDB sink connector#3453
countradooku wants to merge 25 commits into
apache:masterfrom
countradooku:feat/surrealdb-sink-connector

Conversation

@countradooku

@countradooku countradooku commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds a SurrealDB sink connector for writing Iggy messages into SurrealDB over the HTTP API via reqwest.

The connector supports deterministic record IDs, bulk INSERT IGNORE writes for idempotent replay, configurable batch sizing, root/namespace/database/no-auth modes, optional table and offset-index definition, payload modes (auto, json, text, base64/binary), metadata/header/checksum/origin timestamp fields, retry/backoff handling, and runtime metrics logging.

This PR does not depend on the SurrealDB Rust SDK. The v3.1.4 reference is only the Docker image tag used by the integration-test fixture, so the earlier SurrealDB SDK BUSL licensing concern is not applicable to this connector.

It also wires the connector into workspace membership, connector docs, binary artifact builds, edge-release output, version bump scripts, and Docker-backed integration test scaffolding.

Tests

  • cargo fmt --all
  • cargo test -p iggy_connector_surrealdb_sink
  • cargo clippy -p iggy_connector_surrealdb_sink --all-targets -- -D warnings
  • cargo test -p integration --no-run
  • cargo build -p iggy_connector_surrealdb_sink
  • cargo test -p integration -- connectors::surrealdb::surrealdb_sink
  • cargo sort --no-format --workspace --check
  • ./scripts/ci/taplo.sh --check
  • git diff --check

./scripts/ci/markdownlint.sh could not be run locally because markdownlint is not installed in this shell.

@countradooku countradooku marked this pull request as ready for review June 11, 2026 08:49
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jun 11, 2026
@countradooku
feat(connectors): add SurrealDB sink connector
48c3a9d 
SurrealDB is a document database target for Iggy connector users, so the sink writes batches with deterministic record ids and bulk INSERT IGNORE to keep runtime redelivery idempotent without per-message round trips.

Constraint: User explicitly requested the latest SurrealDB Rust SDK and chose to keep it despite BUSL-1.1 license-validation warnings for SurrealDB crates.

Constraint: Local Docker daemon was unavailable, so real-container integration execution could not run here.

Rejected: Per-message SDK writes | too many round trips and weaker batching throughput.

Rejected: Using the testcontainers SurrealDB module | module source hardcodes an older SurrealDB image.

Confidence: medium

Scope-risk: moderate

Directive: Keep record ids deterministic across releases; changing build_record_id breaks replay idempotency.

Tested: cargo fmt --all; cargo sort --no-format --workspace; cargo clippy --all-features --all-targets -- -D warnings; cargo check --all --all-features; cargo test -p iggy_connector_surrealdb_sink; cargo test -p integration --no-run connectors::surrealdb; cargo test --locked --doc; cargo doc --no-deps --all-features --quiet; taplo/license/shellcheck/version/diff/binary checks; prek install

Not-tested: Docker-backed SurrealDB integration execution, because Docker daemon was not running locally.
@countradooku countradooku force-pushed the feat/surrealdb-sink-connector branch from 7b8305a to 48c3a9d Compare June 11, 2026 08:58
@codecov

codecov Bot commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.56%. Comparing base (3b9ba2f) to head (856ab5e).

Additional details and impacted files 
@@             Coverage Diff              @@
##             master    #3453      +/-   ##
============================================
+ Coverage     74.06%   74.56%   +0.49%     
  Complexity      937      937              
============================================
  Files          1249     1243       -6     
  Lines        128257   125075    -3182     
  Branches     104127   100989    -3138     
============================================
- Hits          94996    93261    -1735     
+ Misses        30222    28767    -1455     
- Partials       3039     3047       +8
Components Coverage Δ
Rust Core 75.40% <ø> (+0.69%) ⬆️
Java SDK 62.44% <ø> (ø)
C# SDK 71.40% <ø> (-0.71%) ⬇️
Python SDK 88.88% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 91.22% <ø> (-0.13%) ⬇️
Go SDK 40.14% <ø> (ø)
see 97 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ryerraguntla

ryerraguntla commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 14, 2026
@ryerraguntla

ryerraguntla commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Please check the pre-checks failure

@countradooku

countradooku commented Jun 14, 2026

Copy link
Copy Markdown
Contributor Author

Sure

@countradooku
style(connectors): align SurrealDB license headers
30075ea 
HawkEye maps Rust files to the double-slash license style, so the block comments in the new SurrealDB connector files were treated as missing headers by CI.

Constraint: CI runs the updated HawkEye-based license check with strict header matching.

Confidence: high

Scope-risk: narrow

Tested: PATH=/opt/homebrew/bin:/Users/radudiaconu/.vite-plus/bin:/Users/radudiaconu/.codex/tmp/arg0/codex-arg0uTrL1r:/Users/radudiaconu/Library/pnpm/bin:/Users/radudiaconu/.opencode/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/4.0.0/bin:/Users/radudiaconu/.local/bin:/Users/radudiaconu/Library/Application Support/Herd/bin/:/Users/radudiaconu/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/opt/homebrew/bin:/opt/zerobrew/bin:/Users/radudiaconu/.zerobrew/bin:/Users/radudiaconu/.cargo/bin:/Users/radudiaconu/Library/Application Support/JetBrains/Toolbox/scripts:/Users/radudiaconu/Library/Android/sdk/platform-tools:/Applications/Codex.app/Contents/Resources ./scripts/ci/license-headers.sh --check; cargo fmt --all --check; git diff --check
@countradooku
chore(connectors): remove unused SurrealDB dev dependency
e89b508 
The Rust pre-merge machete job reported that the SurrealDB sink crate declared toml without using it. Removing the dev-dependency is simpler than adding an ignore entry.

Constraint: CI runs cargo machete --with-metadata and fails on unused dependencies.

Rejected: Add cargo-machete metadata ignore | the dependency is genuinely unused.

Confidence: high

Scope-risk: narrow

Tested: cargo sort --no-format --workspace; cargo test -p iggy_connector_surrealdb_sink; PATH=/opt/homebrew/bin:/Users/radudiaconu/.vite-plus/bin:/Users/radudiaconu/.codex/tmp/arg0/codex-arg0uTrL1r:/Users/radudiaconu/Library/pnpm/bin:/Users/radudiaconu/.opencode/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/4.0.0/bin:/Users/radudiaconu/.local/bin:/Users/radudiaconu/Library/Application Support/Herd/bin/:/Users/radudiaconu/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/opt/homebrew/bin:/opt/zerobrew/bin:/Users/radudiaconu/.zerobrew/bin:/Users/radudiaconu/.cargo/bin:/Users/radudiaconu/Library/Application Support/JetBrains/Toolbox/scripts:/Users/radudiaconu/Library/Android/sdk/platform-tools:/Applications/Codex.app/Contents/Resources ./scripts/ci/license-headers.sh --check; cargo fmt --all --check; git diff --check; cargo metadata confirms toml is absent from iggy_connector_surrealdb_sink
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/Cargo.toml Outdated
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
ryerraguntla
ryerraguntla reviewed Jun 14, 2026
View reviewed changes
Comment thread core/integration/tests/connectors/surrealdb/surrealdb_sink.rs
@countradooku

countradooku commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 15, 2026
@ryerraguntla

ryerraguntla commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

/author - could you please check the pre-checks failures

@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Jun 17, 2026
@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 23, 2026
ryerraguntla
ryerraguntla reviewed Jun 24, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
ryerraguntla
ryerraguntla reviewed Jun 24, 2026
View reviewed changes
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
@ryerraguntla

ryerraguntla commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 24, 2026
@hubcio

hubcio commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

how much time do the newly added tests take, on your machine?

@countradooku

countradooku commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

On my machine, the focused SurrealDB integration test filter:
cargo test -p integration -- connectors::surrealdb::surrealdb_sink
runs 4 tests in 1.94s test time, 11.57s wall-clock including incremental compile.
The SurrealDB sink unit tests:
cargo test -p iggy_connector_surrealdb_sink
run 35 tests in 0.02s test time, 0.61s wall-clock.

@countradooku

countradooku commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 24, 2026
hubcio
hubcio requested changes Jun 24, 2026
View reviewed changes

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few non-line notes:

  • the description says this uses the surrealdb rust sdk 3.1.4 and flags BUSL-1.1 license failures from surrealdb sdk crates, but the code uses raw reqwest and the lockfile adds no surrealdb crate. 3.1.4 is the test-container image tag, not a dep, so the BUSL concern is moot. please fix the description and drop that item.
  • the jwt default crypto-provider install (under core/server/src/http/jwt + the a2a_jwt tests) and the cpp-bazel cargo registry retries look unrelated to the surrealdb sink. if the jwt change is a rustls feature-unification fix triggered by linking the sink into the all-crates build, please say so; otherwise split these out.
  • no adversarial round-trip test for the query builder (quotes/brackets in payloads and identifiers). it's safe by construction - identifiers are validated and bodies go through serde_json escaping - but a test would lock that in.

Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/surrealdb_sink/README.md
Comment thread core/connectors/sinks/surrealdb_sink/README.md Outdated
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
Comment thread core/integration/tests/connectors/fixtures/surrealdb/sink.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 24, 2026
@countradooku

countradooku commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 26, 2026
ryankert01
ryankert01 reviewed Jun 27, 2026
View reviewed changes

@ryankert01 ryankert01 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments

Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/surrealdb_sink/src/lib.rs
@ryankert01

ryankert01 commented Jun 27, 2026

Copy link
Copy Markdown
Member

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hubcio hubcio hubcio requested changes
+2 more reviewers
@ryerraguntla ryerraguntla ryerraguntla left review comments
@ryankert01 ryankert01 ryankert01 left review comments
Reviewers whose approvals may not affect merge requirements

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

S-waiting-on-author PR is waiting on author response

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@countradooku @ryerraguntla @hubcio @ryankert01