feat(storage): SQLAlchemy storage provider with PostgreSQL support

[vringar]

Summary

• Rewrite SQLiteStorageProvider to use SQLAlchemy, enabling multi-backend support
• Add PostgreSQL test scenario using pytest-postgresql
• Use BigInteger for columns holding large values (visit_id, request_id, etc.), Integer for bounded values (response_status, duration)
• Conditional imports: SQLite tests work without pytest-postgresql installed
• PostgreSQL fixtures properly gated behind HAS_PYTEST_POSTGRESQL flag

Supersedes #1149. Incorporates fixes from 5 rounds of adversarial review (VDD methodology).

VDD Review History

• Round 1: Found Integer overflow on PostgreSQL, global scenario pollution
• Round 2: Fixed BigInteger, isolated pg scenario — but unconditional import broke SQLite-only envs
• Round 3: Conditional imports, conftest.py fixtures — but gating flag defined and never used
• Round 4: Wired flag to scenarios — but kept wrong psycopg2 variant (abandoned psycopg2-binary)
• Round 5: Correct dependency (psycopg2=2.9.11)
• Final verdict: "...Fine."

Test plan

• SQLite storage tests pass without PostgreSQL installed
• PostgreSQL tests pass when pytest-postgresql is available
• pre-commit run --all-files passes
• No psycopg2-binary references remain

[Copilot]

Pull request overview

Adds a new SQLAlchemy-based structured storage provider and schema definitions to enable multi-backend SQL storage (SQLite + PostgreSQL), updates test scaffolding to include SQLAlchemy/PG scenarios, and adjusts generated test values to fit PostgreSQL INTEGER bounds for certain columns.

Changes:

• Introduces SQLAlchemyStorageProvider (SQLAlchemy Core) and a shared SQLAlchemy schema (TABLE_MAP) for OpenWPM’s structured tables.
• Updates storage test fixtures and adds SQLAlchemy provider tests, including optional PostgreSQL coverage gated on pytest-postgresql.
• Pins/adds PostgreSQL + psycopg2 + SQLAlchemy + pytest-postgresql dependencies; adjusts random test value ranges for duration/response_status.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
openwpm/storage/sqlalchemy_provider.py	New SQLAlchemy-backed StructuredStorageProvider implementation (incl. reflection fallback + coercions).
openwpm/storage/sqlalchemy_schema.py	SQLAlchemy Core table definitions for the structured schema, with BigInteger choices for PG overflow avoidance.
openwpm/storage/sql_provider.py	Rewrites SQLiteStorageProvider into a thin wrapper delegating to SQLAlchemyStorageProvider.
test/storage/test_sqlalchemy_provider.py	New tests for schema equivalence, all-tables insert smoke tests, and _coerce_record behavior; adds optional PG scenario test.
test/storage/fixtures.py	Adds sqlalchemy_sqlite scenario and gated postgresql scenario to structured provider fixtures.
test/storage/conftest.py	Adds conditional pytest-postgresql fixtures for discovery when the dependency is installed.
test/storage/test_values.py	Narrows random ranges for duration/response_status to 32-bit int bounds.
environment.yaml	Adds pinned deps: PostgreSQL, psycopg2, pytest-postgresql, SQLAlchemy.
scripts/environment-unpinned.yaml	Adds corresponding unpinned dependency entries.
.gitignore	Ignores Crosslink-managed local state files.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

store_record rolls back the entire transaction on any insert error. That rollback will also undo any previous successful inserts since the last commit, causing silent data loss for a whole batch/visit when a single bad record arrives. Consider isolating each insert with a SAVEPOINT / nested transaction (e.g., begin_nested()), or committing more frequently and only rolling back the failed statement, while still recovering from PostgreSQL's aborted-transaction state.

Suggested change

	self._connection.execute(sa_table.insert(), record)
	self._sql_counter += 1
	except Exception as e:
	self.logger.error(
	"Unsupported record:\n%s\n%s\ntable=%s\n%s\n"
	% (type(e), e, table, repr(record))
	)
	# On PostgreSQL, a failed statement aborts the entire transaction.
	# All subsequent statements would fail with "InFailedSqlTransaction"
	# until a ROLLBACK is issued. We must rollback here so that
	# subsequent inserts can succeed.
	try:
	self._connection.rollback()
	except Exception as rollback_err:
	self.logger.error("Rollback failed: %s", rollback_err)
	# Isolate each insert in a SAVEPOINT so a single bad record does
	# not roll back earlier successful inserts in the outer transaction.
	# This also recovers cleanly from PostgreSQL's aborted-transaction
	# state by rolling back only the failed statement scope.
	with self._connection.begin_nested():
	self._connection.execute(sa_table.insert(), record)
	self._sql_counter += 1
	except Exception as e:
	self.logger.error(
	"Unsupported record:\n%s\n%s\ntable=%s\n%s\n"
	% (type(e), e, table, repr(record))
	)

[codecov]

Codecov Report

❌ Patch coverage is 96.33028% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.79%. Comparing base (25a7827) to head (29cefcb).

Files with missing lines	Patch %	Lines
openwpm/storage/sqlalchemy_provider.py	96.34%	3 Missing ⚠️
openwpm/storage/sql_provider.py	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1161      +/-   ##
==========================================
+ Coverage   62.36%   62.79%   +0.42%     
==========================================
  Files          40       42       +2     
  Lines        3930     3986      +56     
==========================================
+ Hits         2451     2503      +52     
- Misses       1479     1483       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.