Skip to content

PROTO_VERSIONS.md

Latest commit

 

History

History
239 lines (180 loc) · 7.34 KB

PROTO_VERSIONS.md

File metadata and controls

239 lines (180 loc) · 7.34 KB
Raw

Protocol Buffer Tool Versions

This document specifies the locked Protocol Buffer tools used by Studio's Python code generation and Rust build-time generation.

Note: The Python generator uses the compiler bundled with grpcio-tools; it does not use the system protoc binary. Rust's tonic-prost-build uses the system binary at build time.

Required Tool Versions

Tool Version Purpose
System protoc v30.2 Rust ingestion build-time compiler
grpcio-tools 1.76.0 (bundled libprotoc 31.1) Python protobuf/gRPC code generator

Proto Generation by Language

Python (Backend) - Manual Generation, Committed to Git

Python proto files are:

  • Generated manually via ./scripts/generate_proto.sh
  • Committed to git (backend/app/proto_gen/*.py)
  • Must be regenerated when .proto files change

Why version locking matters: Different protoc versions generate structurally different code. Locking ensures identical code across all environments.

Rust (Ingestion) - Build-time Generation, Not Committed

Rust proto files are:

  • Generated automatically at compile time via build.rs
  • Placed in target/ directory (gitignored)
  • Regenerated fresh on every cargo build

The system compiler is locked to v30.2 so local and CI Rust builds use the same input compiler even though Rust output is not committed.

// ingestion/build.rs
fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_prost_build::configure()
        .compile_protos(
            &["../proto/ingestion.proto", "../proto/auth.proto"],
            &["../proto"],
        )?;
    Ok(())
}

Understanding Version Numbers

What You'll See Where

System compiler used by Rust:

$ protoc --version
libprotoc 30.2

Compiler bundled with the locked Python generator:

$ cd backend
$ uv run python -m grpc_tools.protoc --version
libprotoc 31.1

Generated Python files (header comment):

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: auth.proto
# Protobuf Python Version: 6.31.1  ← Runtime library version (NOT compiler!)

Important Notes

✅ Python files showing Protobuf Python Version: 6.31.1 is CORRECT - this is the protobuf Python runtime library version (from protobuf package), NOT the protoc compiler version

✅ Python runtime version can differ from protoc version - grpcio-tools 1.76.0 bundles libprotoc 31.1 and generates code for protobuf 6.31.1. This is normal.

❌ If system protoc --version is not 30.2, Rust validation is not using the locked compiler.

❌ If uv run python -m grpc_tools.protoc --version is not 31.1, the backend environment does not match the locked Python generator.

Installation Instructions

macOS

Install system protoc v30.2 for Rust builds:

PROTOC_VERSION=30.2
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-osx-x86_64.zip
sudo unzip -o protoc-${PROTOC_VERSION}-osx-x86_64.zip -d /usr/local bin/protoc
sudo unzip -o protoc-${PROTOC_VERSION}-osx-x86_64.zip -d /usr/local 'include/*'
rm protoc-${PROTOC_VERSION}-osx-x86_64.zip

Install Python tools (via uv):

cd backend
uv sync  # Installs grpcio-tools==1.76.0 from uv.lock

Linux

Install system protoc v30.2 for Rust builds:

PROTOC_VERSION=30.2
curl -LO https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-x86_64.zip
sudo unzip -o protoc-${PROTOC_VERSION}-linux-x86_64.zip -d /usr/local bin/protoc
sudo unzip -o protoc-${PROTOC_VERSION}-linux-x86_64.zip -d /usr/local 'include/*'
rm protoc-${PROTOC_VERSION}-linux-x86_64.zip

Install Python tools (via uv):

cd backend
uv sync  # Installs grpcio-tools==1.76.0 from uv.lock

Windows

Install system protoc v30.2 for Rust builds:

$PROTOC_VERSION = "30.2"
Invoke-WebRequest -Uri "https://github.com/protocolbuffers/protobuf/releases/download/v$PROTOC_VERSION/protoc-$PROTOC_VERSION-win64.zip" -OutFile "protoc.zip"
Expand-Archive -Path protoc.zip -DestinationPath "C:\protoc" -Force
# Add C:\protoc\bin to your PATH
Remove-Item protoc.zip

Install Python tools (via uv):

cd backend
uv sync  # Installs grpcio-tools==1.76.0 from uv.lock

Verification

After installation, verify versions:

protoc --version
# Expected: libprotoc 30.2

# For Python (from backend directory with uv environment active)
uv run python -m grpc_tools.protoc --version
# Expected: libprotoc 31.1 (bundled with grpcio-tools 1.76.0)

Regenerating Proto Files

Python (Backend)

cd backend
./scripts/generate_proto.sh

Rust (Ingestion)

No manual regeneration needed. Proto files are generated automatically during:

cargo build
cargo check
cargo run

Automated via Pre-commit Hook

The pre-commit hook automatically regenerates Python proto files before each commit:

  • Regenerates backend/app/proto_gen/*.py files
  • Checks tracked and untracked generated output
  • Stages updated files automatically
  • Prevents commits with stale Python proto code

CI/CD Integration

Python Proto Validation

The root .github/workflows/studio-proto-staleness-check.yml workflow:

  1. Regenerates Python proto files from scratch
  2. Inspects scoped Git porcelain status, including untracked generated files
  3. Fails the build if any differences are detected

Rust Proto Validation

The same workflow runs cargo check in the ingestion directory to verify:

  1. Proto files compile successfully
  2. build.rs correctly references shared proto/ directory

Every CI job that compiles Studio's Rust ingestion target installs system protoc 30.2. The backend and ingestion test jobs download the release archive, verify its pinned SHA-256 checksum, and require the exact libprotoc 30.2 version before compiling.

Troubleshooting

"protoc: command not found"

  • Ensure protoc is in your PATH
  • Verify installation with which protoc

Generated Python code differs from committed files

  • Ensure backend/uv.lock is current and grpcio-tools==1.76.0 is installed
  • Verify the bundled generator reports libprotoc 31.1
  • Regenerate: cd backend && ./scripts/generate_proto.sh

CI validation fails with "Proto files are out of date"

  • Your locked Python environment or grpcio-tools version differs from CI
  • Run uv sync --locked from backend
  • Regenerate proto files locally
  • Commit the updated files

Rust build fails with proto errors

  • Ensure proto/ exists at the Studio root (apps/studio)
  • Check ingestion/build.rs references correct paths
  • Run cargo clean && cargo build to regenerate

Version Update Process

When updating protoc version:

  1. Update version numbers in this file
  2. Update ../../.github/workflows/studio-proto-staleness-check.yml and ../../.github/workflows/studio-backend-tests.yml
  3. Regenerate Python proto files locally
  4. Test that Rust cargo build still works
  5. Commit all updated files in a single commit
  6. Notify developers to update their local installations

References