Skip to content

Commit 7bdf81a

Browse files
committed
Merge branch 'master' into fix/proper-fix-gil-handling
Resolve the conflict in bind/gen_func.go's genFuncBody against master's shimABI refactor (feat: allow opt-in for new backends, #409): - Release the GIL via g.shim.saveThread instead of the inline PyEval_SaveThread/RestoreThread the branch emitted, so the plain-C backends (cffi, pybind11, nanobind), whose shim must not call the CPython API, still emit nothing there. The premarshal pass still runs ahead of it. - Make argCallExpr a pyGen method and have its isSignature case return g.shim.callbackArg(...) rather than sym.py2go. The two agree for the pybindgen and capi backends, but under the plain-C backends sym.py2go is the CPython-API closure from addSignatureType, which wouldn't compile against their preamble.
2 parents e24deba + acffd5c commit 7bdf81a

78 files changed

Lines changed: 5677 additions & 638 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/requirements-capi.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Python packages for the GOPY_BACKEND=capi job in workflows/ci.yml.
2+
# pybindgen is deliberately absent: the capi backend must not need it.
3+
# used by the memory-leak checks on Windows, where the resource module is missing
4+
psutil

‎.github/requirements-cffi.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Python packages for the GOPY_BACKEND=cffi job in workflows/ci.yml.
2+
# pybindgen is deliberately absent: the cffi backend must not need it.
3+
cffi
4+
# used by the memory-leak checks on Windows, where the resource module is missing
5+
psutil

‎.github/requirements-nanobind.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Python packages for the GOPY_BACKEND=nanobind job in workflows/ci.yml.
2+
# pybindgen is deliberately absent: the nanobind backend must not need it.
3+
nanobind
4+
# used by the memory-leak checks on Windows, where the resource module is missing
5+
psutil

‎.github/requirements-pybind11.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Python packages for the GOPY_BACKEND=pybind11 job in workflows/ci.yml.
2+
# pybindgen is deliberately absent: the pybind11 backend must not need it.
3+
pybind11
4+
# used by the memory-leak checks on Windows, where the resource module is missing
5+
psutil

‎.github/workflows/ci.yml‎

Lines changed: 235 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,26 +8,49 @@ on:
88

99
env:
1010
TAGS: "-tags=ci"
11-
COVERAGE: "-coverpkg=github.com/go-python/gopy/..."
1211
# Init() in main_test will make sure all backends are available if
1312
# GOPY_TRAVIS_CI is set
1413
GOPY_TRAVIS_CI: 1
1514
GOTRACEBACK: crash
1615
GO111MODULE: auto
1716

17+
# A new push to a pull request cancels its previous, now outdated, run.
18+
# Pushes to master always run to completion: each one adds to the coverage
19+
# and benchmark history.
20+
concurrency:
21+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
22+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
23+
1824
jobs:
1925

2026
build:
2127
name: Build (${{ matrix.platform }}, Go ${{ matrix.go-version }}, Python ${{ matrix.python-version }})
2228
strategy:
23-
fail-fast: false
2429
matrix:
2530
# TODO: Consider official support matrix (OS and Go versions) and adjust this matrix accordingly
2631
go-version: [1.25.x, 1.24.x, 1.23.x, 1.22.x]
2732
platform: [ubuntu-latest, windows-latest, macos-15]
2833
python-version: ['3.11', '3.12']
2934
runs-on: ${{ matrix.platform }}
35+
# A hung step fails here instead of at GitHub's 6h default.
36+
timeout-minutes: 20
37+
env:
38+
# Coverage is measured on exactly one matrix leg, so the numbers stay
39+
# comparable from commit to commit.
40+
COVERAGE_LEG: ${{ matrix.platform == 'ubuntu-latest' && matrix.go-version == '1.25.x' && matrix.python-version == '3.12' }}
3041
steps:
42+
# Real-time scanning inspects every file the compilers, linkers and pip
43+
# write, which slows Windows jobs down. Turned off first, so checkout and
44+
# the Go/Python installs benefit too. The status is printed before and
45+
# after so the effect can be checked in the log.
46+
- name: Disable Windows Defender real-time scanning
47+
if: runner.os == 'Windows'
48+
shell: pwsh
49+
run: |
50+
Get-MpComputerStatus | Select-Object RealTimeProtectionEnabled
51+
Set-MpPreference -DisableRealtimeMonitoring $true
52+
Get-MpComputerStatus | Select-Object RealTimeProtectionEnabled
53+
3154
- name: Checkout code
3255
uses: actions/checkout@v4
3356

@@ -45,36 +68,59 @@ jobs:
4568
- name: Install Linux packages
4669
if: matrix.platform == 'ubuntu-latest'
4770
run: |
48-
sudo apt-get update
49-
sudo apt-get install curl libffi-dev python3-cffi python3-pip
5071
# install pybindgen
5172
python3 -m pip install --user -U pybindgen
5273
# install goimports
53-
go install golang.org/x/tools/cmd/goimports@latest
74+
go install golang.org/x/tools/cmd/goimports@v0.29.0
5475
5576
- name: Install macOS packages
5677
if: startsWith(matrix.platform, 'macos-')
5778
run: |
5879
python3 -m pip install -U pybindgen
59-
go install golang.org/x/tools/cmd/goimports@latest
80+
go install golang.org/x/tools/cmd/goimports@v0.29.0
6081
6182
- name: Install Windows packages
6283
if: matrix.platform == 'windows-latest'
6384
run: |
6485
# install pybindgen and psutil (for memory tracking in tests)
6586
python -m pip install -U pybindgen psutil
6687
# install goimports
67-
go install golang.org/x/tools/cmd/goimports@latest
88+
go install golang.org/x/tools/cmd/goimports@v0.29.0
6889
6990
7091
- name: Build-Linux
7192
if: matrix.platform == 'ubuntu-latest'
7293
run: |
7394
make
7495
- name: Test Linux
75-
if: matrix.platform == 'ubuntu-latest'
96+
if: matrix.platform == 'ubuntu-latest' && env.COVERAGE_LEG != 'true'
7697
run: |
7798
make test
99+
- name: Test Linux (with coverage)
100+
if: env.COVERAGE_LEG == 'true'
101+
run: |
102+
make cover
103+
- name: Coverage summary
104+
if: env.COVERAGE_LEG == 'true'
105+
run: |
106+
{
107+
echo "## Test coverage"
108+
echo
109+
echo "Commit \`${GITHUB_SHA}\`: **$(go tool cover -func=coverage.out | tail -n 1 | awk '{print $NF}')** of statements"
110+
echo
111+
echo "<details><summary>Per-function coverage</summary>"
112+
echo
113+
echo '```'
114+
go tool cover -func=coverage.out
115+
echo '```'
116+
echo "</details>"
117+
} >> "$GITHUB_STEP_SUMMARY"
118+
- name: Upload coverage profile
119+
if: env.COVERAGE_LEG == 'true'
120+
uses: actions/upload-artifact@v4
121+
with:
122+
name: coverage
123+
path: coverage.out
78124

79125
- name: Build-macOS
80126
if: startsWith(matrix.platform, 'macos-')
@@ -93,6 +139,185 @@ jobs:
93139
if: matrix.platform == 'windows-latest'
94140
run: |
95141
go test -v ./...
142+
# Long-term history/trends. Needs a CODECOV_TOKEN repo secret (or
143+
# tokenless uploads enabled for the org on codecov.io); without one this
144+
# step only warns, and the step summary + artifact above still work.
96145
- name: Upload-Coverage
97-
if: matrix.platform == 'ubuntu-latest'
98-
uses: codecov/codecov-action@v4
146+
if: env.COVERAGE_LEG == 'true'
147+
uses: codecov/codecov-action@v5
148+
with:
149+
files: coverage.out
150+
disable_search: true
151+
token: ${{ secrets.CODECOV_TOKEN }}
152+
fail_ci_if_error: false
153+
154+
# Builds and tests each opt-in backend (GOPY_BACKEND), beside the main
155+
# matrix and on one Go version. None installs pybindgen: no backend here
156+
# may need it.
157+
# cffi: loads the cgo shim at runtime, with no C compiler step.
158+
# pybind11: compiles a C++ module against the same shim (see plainCShim in
159+
# bind/plainc.go), so needs no extra system packages: a C++
160+
# compiler is already required for CGO_ENABLED=1 everywhere.
161+
# nanobind: as pybind11 (see buildCXXModule in cmd_build.go), and also
162+
# compiles nanobind's own runtime (nb_combined.cpp) into every
163+
# module.
164+
# capi: builds exactly as the default backend does, except that
165+
# build.py writes the C module itself (see bind/capi_build.py).
166+
backends:
167+
name: ${{ matrix.backend }} backend (${{ matrix.platform }}, Python ${{ matrix.python-version }})
168+
strategy:
169+
matrix:
170+
backend: [cffi, pybind11, nanobind, capi]
171+
platform: [ubuntu-latest, windows-latest, macos-15]
172+
python-version: ['3.11', '3.12']
173+
include:
174+
# CXX defaults to "c++" in buildCXXModule (cmd_build.go), which
175+
# isn't a recognized command on windows-latest's MinGW toolchain (the
176+
# same one CGO_ENABLED=1 already needs there, so no extra install is
177+
# needed). timeout raises go test's own default (10m): each test
178+
# here compiles twice (cgo, then a separate C++ step), and once
179+
# measured at ~18.5s/test average on windows-latest, 32 tests alone
180+
# used 590s of the default budget.
181+
- backend: pybind11
182+
cxx: g++
183+
timeout: 30m
184+
- backend: nanobind
185+
cxx: g++
186+
timeout: 30m
187+
runs-on: ${{ matrix.platform }}
188+
# Above the longest go test -timeout (30m), so that go test's own timeout,
189+
# with its stack dump, fires first.
190+
timeout-minutes: 40
191+
env:
192+
GOPY_BACKEND: ${{ matrix.backend }}
193+
CXX: ${{ matrix.cxx }}
194+
# Each backend also uploads coverage from one leg: much of bind/ (and
195+
# cmd_build.go) runs only under a given backend. Codecov merges these
196+
# with the build job's (default-backend) upload for the same commit.
197+
COVERAGE_LEG: ${{ matrix.platform == 'ubuntu-latest' && matrix.python-version == '3.12' }}
198+
# print the python stack if the process crashes, e.g. at exit
199+
PYTHONFAULTHANDLER: 1
200+
steps:
201+
# Real-time scanning inspects every file the compilers, linkers and pip
202+
# write, which slows Windows jobs down. Turned off first, so checkout and
203+
# the Go/Python installs benefit too. The status is printed before and
204+
# after so the effect can be checked in the log.
205+
- name: Disable Windows Defender real-time scanning
206+
if: runner.os == 'Windows'
207+
shell: pwsh
208+
run: |
209+
Get-MpComputerStatus | Select-Object RealTimeProtectionEnabled
210+
Set-MpPreference -DisableRealtimeMonitoring $true
211+
Get-MpComputerStatus | Select-Object RealTimeProtectionEnabled
212+
213+
- name: Checkout code
214+
uses: actions/checkout@v4
215+
216+
- name: Set up Python
217+
uses: actions/setup-python@v5
218+
with:
219+
python-version: ${{ matrix.python-version }}
220+
cache: pip
221+
cache-dependency-path: .github/requirements-${{ matrix.backend }}.txt
222+
223+
- name: Install Go
224+
uses: actions/setup-go@v5
225+
with:
226+
go-version: 1.25.x
227+
cache: true
228+
229+
- name: Install packages
230+
run: |
231+
python -m pip install -r .github/requirements-${{ matrix.backend }}.txt
232+
go install golang.org/x/tools/cmd/goimports@v0.29.0
233+
234+
- name: Build
235+
run: go build -v ./...
236+
237+
- name: Test
238+
if: env.COVERAGE_LEG != 'true'
239+
run: go test -v -timeout=${{ matrix.timeout || '10m' }} ./...
240+
241+
- name: Test (with coverage)
242+
if: env.COVERAGE_LEG == 'true'
243+
run: make cover
244+
245+
- name: Upload-Coverage
246+
if: env.COVERAGE_LEG == 'true'
247+
uses: codecov/codecov-action@v5
248+
with:
249+
files: coverage.out
250+
disable_search: true
251+
token: ${{ secrets.CODECOV_TOKEN }}
252+
fail_ci_if_error: false
253+
254+
# Compares per-call overhead across backends (see _examples/bench/run.sh):
255+
# not a pass/fail check, just a table uploaded as a build artifact. Runs
256+
# on ubuntu-latest only -- the comparison is between backends, not OSes.
257+
benchmark:
258+
name: benchmark backends
259+
runs-on: ubuntu-latest
260+
timeout-minutes: 15
261+
steps:
262+
- name: Checkout code
263+
uses: actions/checkout@v4
264+
265+
- name: Set up Python
266+
uses: actions/setup-python@v5
267+
with:
268+
python-version: '3.12'
269+
270+
- name: Install Go
271+
uses: actions/setup-go@v5
272+
with:
273+
go-version: 1.25.x
274+
cache: true
275+
276+
- name: Install packages
277+
run: |
278+
python -m pip install pybindgen cffi pybind11 nanobind
279+
go install golang.org/x/tools/cmd/goimports@v0.29.0
280+
281+
- name: Run benchmark
282+
run: |
283+
bash _examples/bench/run.sh | tee benchmark.txt
284+
python _examples/bench/bench_json.py < benchmark.txt > benchmark.json
285+
286+
- name: Upload benchmark table
287+
uses: actions/upload-artifact@v4
288+
with:
289+
name: benchmark
290+
path: |
291+
benchmark.txt
292+
benchmark.json
293+
294+
# The results of earlier runs on master, kept in the Actions cache (no
295+
# branch or token needed): every run restores the latest, a pull request
296+
# from its base branch's.
297+
- name: Restore benchmark history
298+
uses: actions/cache/restore@v4
299+
with:
300+
path: benchmark-history
301+
key: benchmark-history-${{ github.run_id }}
302+
restore-keys: benchmark-history-
303+
304+
# Compares with the latest master run in the job summary, warning (but
305+
# never failing, since CI runners' timings are noisy) at 2x slower, and
306+
# adds this run to the history.
307+
- name: Compare with earlier runs
308+
uses: benchmark-action/github-action-benchmark@v1
309+
with:
310+
name: gopy backends
311+
tool: customSmallerIsBetter
312+
output-file-path: benchmark.json
313+
external-data-json-path: benchmark-history/data.json
314+
summary-always: true
315+
fail-on-alert: false
316+
317+
# Only master's runs make up the history.
318+
- name: Save benchmark history
319+
if: github.event_name == 'push'
320+
uses: actions/cache/save@v4
321+
with:
322+
path: benchmark-history
323+
key: benchmark-history-${{ github.run_id }}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: GoReleaser Check
2+
3+
# Validates .goreleaser.yaml against a real snapshot build (cross-compile +
4+
# archive generation, no publish) so config/toolchain regressions surface on
5+
# the PR instead of after release-please has already cut a tag & GitHub
6+
# Release. See release-please.yml's `publish` job, which runs the real
7+
# `goreleaser release --clean` only after that release already exists.
8+
9+
on:
10+
pull_request:
11+
branches: [ '**' ]
12+
paths:
13+
- '.goreleaser.yaml'
14+
- '.github/workflows/goreleaser-check.yml'
15+
- '.github/workflows/release-please.yml'
16+
- 'go.mod'
17+
- 'go.sum'
18+
- 'version.go'
19+
workflow_dispatch:
20+
21+
jobs:
22+
snapshot:
23+
runs-on: ubuntu-latest
24+
steps:
25+
- name: Checkout code
26+
uses: actions/checkout@v4
27+
with:
28+
fetch-depth: 0
29+
30+
- name: Install Go
31+
uses: actions/setup-go@v5
32+
with:
33+
go-version-file: go.mod
34+
cache: true
35+
36+
- name: Validate GoReleaser config
37+
uses: goreleaser/goreleaser-action@v6
38+
with:
39+
distribution: goreleaser
40+
version: latest
41+
args: check
42+
43+
- name: Snapshot build (validates cross-compilation & archives)
44+
uses: goreleaser/goreleaser-action@v6
45+
with:
46+
distribution: goreleaser
47+
version: latest
48+
args: release --snapshot --clean

0 commit comments

Comments
 (0)