Question: how should property-based tests be accepted without making CI flaky?

[kumagi]

Operating System

• Windows
• macOS
• Linux
• FreeBSD
• NetBSD
• Android
• iOS

What feature would you like to be added?

I would like to ask how the maintainers feel about property-based tests (PBTs) for purego, and if they are welcome, in what form they should be contributed so that CI does not become flaky.

What I mean by PBT here: instead of one hand-written case with one fixed signature, a test generates many random inputs/types from a small model, runs them against an oracle, and asserts an invariant that must hold for every generated case. The oracle in my prototypes is the real C ABI itself: the test emits C declarations equivalent to the generated Go types, compiles them into a shared library, and compares purego's behavior byte-for-byte against what the C compiler did. Generation is seeded so failures reproduce deterministically, and every failure dumps the seed, the Go type, the emitted C source, and a hex diff.

I prototyped three such probes locally:

1. internal/strings CString / GoString round trip. Generates 5000 random strings over a mixed alphabet (ASCII, symbols, multibyte runes, sometimes NUL-terminated to cover both the zero-copy and copy paths in CString) plus deterministic edge cases (empty, leading/interior NULs, 64KB strings, 4096 Japanese characters). Invariant: GoString(CString(s)) equals the prefix of s up to the first NUL, i.e. the C-string contract survives both paths untouched.

2. RegisterFunc struct ABI round trip. Generates random struct types (scalars, small arrays, nested structs) and call signatures mixing structs with scalars, emits equivalent C declarations, compiles them, and round-trips values through the real C ABI in both argument and return directions, including register-exhaustion and stack layouts. Invariant: every argument byte the Go side passes must arrive intact, and every return byte must come back intact.

3. NewCallback argument/return round trip. Generates random callback signatures (scalar/array/struct parameters with scalar/struct/void returns), emits a C forwarder that calls the Go callback through the real C ABI, and checks both directions byte-for-byte. Invariant: arguments passed from Go must reach the Go callback uncorrupted, and the callback's return value must come back uncorrupted. The current prototype skips Windows (stdcall restrictions on callback argument types) and excludes float returns per compileCallback; I am not sure how far the Windows boundary should be covered, hence all OS boxes are checked above.

The expected benefit for purego, if these are accepted in some form: struct and callback paths encode per-OS/per-arch ABI rules (register classes, eightbyte classification, padding, memory-vs-register decisions) that are easy to get subtly wrong and hard to cover with hand-written cases. A seeded random round trip against the C compiler acts as a conformance net — it catches drift and corner cases a fixed table misses, and each counterexample is directly actionable because it carries its repro (seed + C source + diff).

My concern, and the reason this is a question rather than a PR: while the failures such tests find are legitimate, landing randomized tests directly into the normal unit-test suite makes CI noisy. So I would like to ask:

1. Are property-based / randomized round-trip tests welcome in this repository at all?
2. If a PBT is red on current main but green after a fix, how should it be submitted — together with the fix, held until the fix lands, landed as skipped, or some other way?
3. Where should such tests live: normal *_test.go run by go test ./..., files behind a build tag, or a separate workflow (e.g. nightly / manual)?
4. What determinism/reproducibility rules would you want: fixed seeds, logged seeds, C-source dump on failure, byte-level diff output?
5. Is there interest in Go native fuzz (FuzzXxx + seed corpus in CI, longer fuzzing locally/nightly) as an alternative shape for these tests?

Why is this needed?

As background for the questions above, I ran the three prototypes locally on Linux/amd64 (Go 1.26.x, main at 763bb99):

1. CString / GoString round trip: green on current main.

2. RegisterFunc struct ABI round trip: 27 counterexamples on current main. All fall into amd64 struct handling, for example: fields packed back-to-back ignoring in-memory offsets (a 64-bit field overwriting pending small fields and shifting later arguments by one slot; padding ignored, e.g. int32 at offset 4 in {int8; int32}); a struct split between registers and stack when one register class is exhausted while the SysV ABI requires the whole aggregate to go in memory; struct returns reconstructed from hand-rolled per-field checks instead of per-eightbyte classification (e.g. struct{ [1]float32 } read from RAX while the callee delivers it in xmm0, float[3] followed by uint16 panicking, nested all-float first eightbyte reading both halves from xmm0). The test is deterministic (fixed seeds) and turns green once those amd64 paths are fixed.

3. NewCallback argument/return round trip: 20 counterexamples on current main. Analysis showed the same amd64 struct-packing cluster manifesting in the C-to-Go direction: where GCC correctly passes a struct argument in two words, the purego sender side packed it into one word ignoring offsets. During development this test even produced a hard crash (SIGSEGV), which shows its detection power. It also turns green once the amd64 paths are fixed.

I am not submitting these tests or fixes here yet because of the CI-noise concern above. I can share minimal reproducers or the prototype tests if that helps discussion — please let me know what form would be most useful.

[kumagi]

Small-start proposal: fuzz only CString/GoString first

Following up on the questions above, I'd like to propose starting small:
land a single Go-native fuzz test for the CString/GoString round trip,
plus a nightly workflow that runs it. The struct/callback probes would stay
as deterministic Test... prototypes for now (a Fuzz prefix would actually
break them: the toolchain requires func FuzzXxx(*testing.F) and fails a
Fuzz-named test that never calls f.Fuzz, so non-fuzz PBTs must keep the
Test name).

Proposed test (internal/strings/strings_fuzz_test.go):

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: 2026 The Ebitengine Authors

package strings

import (
	"strings"
	"testing"
	"unsafe"
)

// FuzzCStringGoString fuzzes the CString -> GoString round trip.
//
// Whatever CString does - zero-copy for a string that already ends in NUL,
// copy otherwise - the bytes must survive untouched, and GoString must return
// exactly the prefix of the input up to its first NUL, which is the contract
// of a C string conversion.
func FuzzCStringGoString(f *testing.F) {
	// Seed corpus: deterministic edge cases. These run on every plain
	// `go test`, so the fast deterministic checks stay in normal CI.
	for _, s := range []string{
		"",
		"\x00",
		"a",
		"\x00\x00",
		"\x00a\x00",
		"hello",
		"hello\x00",
		"a\x00b",
		"日本語",
		"日本語\x00後",
		strings.Repeat("x", 1024),
		strings.Repeat("x", 1024) + "\x00",
	} {
		f.Add(s)
	}
	f.Fuzz(func(t *testing.T, s string) {
		want := s
		if i := strings.IndexByte(s, 0); i >= 0 {
			want = s[:i]
		}
		if got := GoString(uintptr(unsafe.Pointer(CString(s)))); got != want {
			t.Fatalf("round trip of %q: got %q, want %q", s, got, want)
		}
	})
}

Why this follows the Go fuzzing conventions exactly:

• func FuzzCStringGoString(f *testing.F) in a *_test.go file, exactly one
  f.Fuzz call, fuzz argument is string (one of the allowed types), seeds
  added via f.Add with matching types.
• No build tag or separate directory needed: plain go test runs only the
  seeds (fast, deterministic), and fuzzing happens only under
  go test -fuzz=FuzzCStringGoString.
• A failing input found while fuzzing is minimized and written to
  testdata/fuzz/FuzzCStringGoString/, after which plain go test replays
  it as a regression test automatically.

What I verified locally (linux/amd64, Go 1.26.x):

• go test ./internal/strings/ -count=1: all 12 seeds pass.
• go test -fuzz=FuzzCStringGoString -fuzztime=20s: ~40M execs
  (~2M/sec), no counterexamples. testdata/fuzz was not created, as expected
  when nothing fails. (This is evidence of health, not a proof of absence.)
• go vet, gofmt -s -l, go test -race, and
  go test "-gcflags=all=-N -l" are all clean for this package.

One precision on "notice breakage immediately": there are two layers here.
Normal CI catches regressions inside the 12 seeds immediately on the breaking
PR. Breakage outside the seed range is found by the nightly fuzz run, and once
found, its minimized input becomes a permanent seed, so the next normal CI
run catches that shape forever. That is the loop that makes this useful.

Why this one first:

• Pure Go, no C toolchain: runs on every OS/arch in normal CI, unlike the
  struct/callback probes that need to compile a .so.
• Fast (~2M execs/sec), so nightly -fuzztime buys real exploration.
• Fully visible to coverage guidance: the code under test is plain Go
  (hasSuffix branch for the zero-copy vs copy paths, the NUL scan loop in
  GoString). The struct/callback paths bottom out in .s files and the C
  compiler's decisions, which coverage cannot see, so they need a different
  (generational, type-shape) design and should come later.

Nightly sketch (separate workflow, normal CI untouched):

name: Nightly PBT
on:
  schedule:
    - cron: '17 0 * * *' # off the top of the hour: less delay/drop risk
  workflow_dispatch:
permissions:
  contents: read
  issues: write
jobs:
  pbt:
    strategy:
      matrix:
        os: [ubuntu-latest, ubuntu-24.04-arm]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-go@v6
        with:
          go-version: '1.27.x'
      - run: go test -fuzz=FuzzCStringGoString -fuzztime=10m ./internal/strings/
      - name: File or update issue on failure
        if: failure()
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # de-duplicate: comment on the open nightly-pbt issue if one exists,
          # otherwise create it; attach the run link + log tail + testdata artifact
          ...

permissions: issues: write is required for the gh issue create step;
gh is preinstalled on the runners so no third-party action is needed. The
... part is the de-dup logic (search open nightly-pbt issues first), so a
red night doesn't spam a new issue every day.

If this direction sounds good, I'm happy to open a PR with just these two
pieces (the fuzz test + the nightly workflow) as the small start. The
struct/callback work would then follow as deterministic Test + minimized
regression cases, not as Fuzz tests. Please let me know.

[hajimehoshi]

I'm fine we have such fuzzing tests.

For issues: write, can we see the fuzzing test results for a while first? If this alerts too many problems, we have to reduce them first, or the issues would be spammed.

@TotallyGamerJet Thoughts?

[kumagi]

Thanks, agreed — let's not add issues: write yet.

To clarify the spam surface of this specific PR: it contains only FuzzCStringGoString, which is currently green locally (~40M execs, no counterexamples), so it should be quiet. The noisy ones are the struct/callback probes, which are out of scope here and whose known failure clusters are already tracked as #520, #522, #525, #526, #527 — the nightly job would not re-report those.

How about this phasing:

1. First PR: just the fuzz test + the nightly workflow without the issue-filing step (no issues: write). Failures show up as red Actions runs with the log tail and testdata/fuzz artifact. We watch that for a while.
2. Only if the signal looks good, a follow-up PR adds de-duplicated issue filing (nightly-pbt label, comment-on-open-issue instead of new issue per night).

If that sounds good I'll open the Phase 1 PR in this shape.

[hajimehoshi]

First PR: just the fuzz test + the nightly workflow without the issue-filing step (no issues: write). Failures show up as red Actions runs with the log tail and testdata/fuzz artifact. We watch that for a while.

Will this send emails to the maintainers?

[kumagi]

Good question — I checked the docs (https://docs.github.com/en/actions/concepts/workflows-and-actions/notifications-for-workflow-runs). For scheduled workflows, GitHub sends failure notifications only to a single user: whoever created the workflow (retargeted to whoever last touched the cron syntax, or re-enabled it). So no, a red nightly would not email all maintainers; only the workflow author gets the email, and everyone else would have to look at the Actions tab.

Given Phase 1 is explicitly a watch-and-see period, I think that's actually fine: the signal observer is me, and I'll report back here periodically (or on any red). If the signal proves good, Phase 2 (de-duplicated issue filing) is what gives all maintainers push notifications via their normal issue-watch settings.

So the proposal stands: Phase 1 PR = fuzz test + nightly workflow, no issues: write, notification goes to the workflow author only. Does that work for you?