Automated Prow CI failure analysis using Red Hat's ship-help MCP (AI helpdesk).
- CLI tool: Analyze any Prow job URL from the command line
- Slack bot: Auto-analyzes Prow URLs posted in monitored Slack channels
- Comprehensive analysis: Uses ship-help MCP with access to:
- Jira issues (active, historical, related)
- GitHub repositories and PRs
- Build logs and artifacts
- Test results and history
- Firewatch automated triage
- Slack team discussions
- Internal documentation
- Historical failure patterns
# Set environment variables
export SHIP_HELP_MCP_URL="https://ship-help-mcp-continuous-release-tooling--ship-help-bot.apps.gpc.ocp-hub.prod.psi.redhat.com/personas/ocp_ai_helpdesk/mcp"
export SHIP_HELP_MCP_TOKEN="your-jwt-token"
# Build
go build ./cmd/prow-analyzer
# Analyze a job
./prow-analyzer analyze https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-stolostron-policy-collection-main-ocp4.22-interop-opp-aws/2066255424226594816# Additional environment variables for Slack
export SLACK_BOT_TOKEN="xoxb-..."
export SLACK_APP_TOKEN="xapp-..."
export MONITORED_CHANNELS="C12345678,C87654321"
# Build and run
go build ./cmd/prow-analyzer-bot
./prow-analyzer-botThe bot monitors configured Slack channels and automatically analyzes any Prow URLs posted, replying in a thread with:
- Root cause analysis
- Related Jira issues
- Recurring pattern detection
- Recommended actions
git clone https://github.com/oramraz/prow-analyzer.git
cd prow-analyzer
go mod download
go build ./cmd/prow-analyzer
go build ./cmd/prow-analyzer-bot# Build and push image
podman build -t quay.io/oramraz/prow-analyzer-bot:latest .
podman push quay.io/oramraz/prow-analyzer-bot:latest
# Update secrets in deploy/openshift/deployment.yaml
# Then deploy:
oc apply -f deploy/openshift/deployment.yamlRequired:
SHIP_HELP_MCP_URL- Ship-help MCP endpointSHIP_HELP_MCP_TOKEN- Authentication token for ship-help MCP
Slack bot only:
SLACK_BOT_TOKEN- Slack bot token (xoxb-...)SLACK_APP_TOKEN- Slack app token for socket mode (xapp-...)MONITORED_CHANNELS- Comma-separated list of channel IDs
Ship-help MCP Token:
- Contact #ship-users in Slack or
- Use existing token from ship-help-bot access
Slack Tokens:
- Create a Slack app at https://api.slack.com/apps
- Enable Socket Mode and Events API
- Required bot scopes:
channels:history,chat:write,app_mentions:read - Install to workspace and get tokens from OAuth & Permissions
-
pkg/analyzer - Core MCP client
- Session initialization and management
- MCP protocol handling (JSON-RPC over SSE)
- Prow URL extraction and validation
- Response formatting
-
cmd/prow-analyzer - CLI tool
- Single job analysis
- Standalone operation
- Human-readable output
-
cmd/prow-analyzer-bot - Slack bot
- Real-time channel monitoring
- Automatic URL detection
- Threaded responses
- Socket mode for reliable event delivery
-
pkg/slack/handler - Event processing
- Message filtering
- Channel authorization
- Async analysis execution
1. Initialize session → Get session ID
2. Call ask_persona with Prow URL
3. Parse SSE response stream
4. Extract analysis from JSON-RPC result
🔍 Prow Analyzer Analysis
**Root Cause:**
Test failure in openshift-tests due to timeout waiting for operator rollout.
**Related Jira Issues:**
- OCPBUGS-12345: Operator timeout in e2e tests (Open)
- OCPBUGS-11111: Similar timeout resolved by increasing wait time (Closed)
**Recurring Patterns:**
This failure pattern appears in 3 other jobs in the last 7 days, all in
the same test suite.
**Recommendations:**
1. Check cluster operator status at test start
2. Increase timeout from 5m to 10m
3. Review OCPBUGS-12345 for ongoing investigation
Analysis completed in 78.6s • Powered by ship-help MCP
Based on the working implementation from:
- openshift/ci-tools PR #5251 (analyzer code)
- openshift/release PR #80559 (configuration)
- Credit to @chaclark1974 for original Slack integration design
Apache 2.0