fix: lazy import integrations to avoid pyspark bundling error and add deep research (#1)

* fix: lazy import integrations to avoid pyspark bundling error

The CLI binary was failing with "No such file or directory:
error-conditions.json" because pyspark was being eagerly imported
when any integration was accessed.

Changes:
- Make integrations/__init__.py use lazy imports via __getattr__
- Update PyInstaller spec to use correct integration paths
- Remove library-only integrations (polars, duckdb, spark) from bundle
- Only bundle integrations with deploy capability (bigquery, snowflake)

* feat: add --output option to search and extract commands

* feat: add deep research command to CLI

Add `parallel-cli research` command group for running deep research tasks:
- `research run` - Create and run research tasks with polling
- `research status` - Check task status
- `research poll` - Resume polling existing tasks
- `research processors` - List available processor tiers

Also includes:
- New core/research.py module with research API functions
- Comprehensive tests with mocking (36 tests)
- Refactored shared polling logic to reduce duplication
- Added write_json_output() helper for consistent output handling

* fix: separate markdown content from JSON metadata in research output

* fix: update processor latencies to match documentation

* fix: use TextSchemaParam to get markdown output from deep research

* bump version and research updates

* json

* fix: add type assertions for pyrefly type checker

Add assertions after validation to narrow optional types to non-None,
fixing 10 pyrefly bad-argument-type errors in enrich_run and enrich_plan.

* fix: resolve pyrefly type checking errors

- Add explicit urllib.error import in auth.py
- Use SDK BetaRunInputParam type in batch.py and spark/streaming.py
- Add explicit type annotations to fix type inference issues
- Use row.asDict() for proper PySpark Row field access
- Add type narrowing assertion in test_cli.py
- Exclude non-core files from pyrefly (cloud_function, examples, notebooks, scripts)
- Add pyrefly pre-commit hook (local, uses uv run)

Author: parallel-m

.pre-commit-config.yaml

@@ -14,3 +14,11 @@ repos:
       - id: ruff-check
       # Run the formatter.
       - id: ruff-format
+  - repo: local
+    hooks:
+      - id: pyrefly-check
+        name: pyrefly (type checking)
+        entry: uv run pyrefly check
+        language: system
+        pass_filenames: false
+        types: [python]

parallel-web-tools.spec

@@ -36,24 +36,18 @@ a = Analysis(
         'parallel_web_tools.cli',
         'parallel_web_tools.cli.commands',
         'parallel_web_tools.cli.planner',
-        # Processors
+        # Processors (for local file/db enrichment)
         'parallel_web_tools.processors',
         'parallel_web_tools.processors.csv',
         'parallel_web_tools.processors.duckdb',
         'parallel_web_tools.processors.bigquery',
-        # Integrations
-        'parallel_web_tools.polars',
-        'parallel_web_tools.polars.enrich',
-        'parallel_web_tools.duckdb',
-        'parallel_web_tools.duckdb.batch',
-        'parallel_web_tools.duckdb.udf',
-        'parallel_web_tools.bigquery',
-        'parallel_web_tools.bigquery.deploy',
-        'parallel_web_tools.snowflake',
-        'parallel_web_tools.snowflake.deploy',
-        'parallel_web_tools.spark',
-        'parallel_web_tools.spark.udf',
-        'parallel_web_tools.spark.streaming',
+        # Integrations with deploy capability (bundled in CLI)
+        'parallel_web_tools.integrations',
+        'parallel_web_tools.integrations.bigquery',
+        'parallel_web_tools.integrations.bigquery.deploy',
+        'parallel_web_tools.integrations.snowflake',
+        'parallel_web_tools.integrations.snowflake.deploy',
+        # Note: polars, duckdb, spark integrations are library-only (no deploy step)
         # Dependencies that might not be auto-detected
         'click',
         'questionary',
@@ -65,7 +59,6 @@ a = Analysis(
         'parallel',
         'duckdb',
         'sqlalchemy',
-        'polars',
         'pandas',
         'certifi',
     ],

parallel_web_tools/__init__.py

@@ -27,7 +27,7 @@
     run_tasks,
 )
 
-__version__ = "0.0.2"
+__version__ = "0.0.3"
 
 __all__ = [
     # Auth