Skip to content
Nagasai-DhanushPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CI Agent - LLM-Powered CI/CD Monitoring & Autonomous Remediation

A sophisticated CI/CD failure monitoring agent that uses LLM-powered analysis to diagnose build failures and apply safe, autonomous remediation actions. Integrated with Featherless AI and Bright Data for enhanced intelligence.

🎯 Key Features

  • 🤖 LLM-Powered Error Classification: Uses Featherless AI with chain-of-thought reasoning to analyze CI/CD failures
  • 🔧 Autonomous Remediation: Detects root cause and applies appropriate fixes from a safe playbook
  • 🛡️ Safety Guardrails: Confidence thresholds, retry limits, and validation to prevent unsafe actions
  • 📊 Bright Data Enrichment: Cross-references errors with external signals and known issue database
  • 📌 Human Feedback Loop: Notifies humans AFTER resolution, not before
  • 📈 Learning & Monitoring: Tracks success rates and patterns to improve over time
  • 🎨 Rich Slack Integration: Explainable summaries with reasoning steps and metrics

📋 Architecture Overview

CI Pipeline Failure
        ↓
  GitLab Webhook
        ↓
  Error Log Extraction
        ↓
  Bright Data Enrichment (external signals)
        ↓
  LLM Analysis (chain-of-thought reasoning)
        ↓
  Confidence & Safety Checks
        ↓
  Playbook Action Selection
        ↓
  Autonomous Execution
        ↓
  Slack Notification (AFTER fixing)
        ↓
  Monitoring & Analytics

🔍 Error Classification Types

The system identifies and handles:

  1. Flaky Tests - Transient test failures → Retry
  2. Dependency Issues - Package conflicts or missing deps → Reinstall/Clear Cache
  3. Environment Issues - Network/connectivity problems → Retry
  4. Timeout - Operations exceeding time limits → Retry with backoff
  5. Rate Limit - External service quota exceeded → Backoff & Retry
  6. Permission Error - Auth/credential issues → Manual review
  7. Code Error - Application logic errors → Manual review

🚀 Quick Start

Prerequisites

  • Node.js 18+
  • npm dependencies
  • GitLab Personal Access Token
  • Slack Webhook URL
  • Featherless AI API Key
  • Bright Data API Key (optional)

Installation

# Clone & install
npm install

# Setup environment
cp .env.example .env
# Edit .env with your credentials

Configuration

Create a .env file:

# Server
PORT=3000
NODE_ENV=development

# GitLab
GITLAB_TOKEN=glpat_xxxxxxxxxxxxx
GITLAB_API_BASE=https://gitlab.com/api/v4

# Slack
SLACK_WEBHOOK=https://hooks.slack.com/services/...

# AI
FEATHERLESS_API_KEY=ftr_xxxxxxxxxxxxx

# Enrichment
BRIGHTDATA_API_KEY=xxxxxxxxxxxxx

Running

# Development
node server.js

# With logging
DEBUG=* node server.js

# Production
NODE_ENV=production node server.js

🔌 GitLab Setup

  1. Create a Personal Access Token in GitLab (Settings → Access Tokens)

    • Scopes: api, read_api
  2. Add webhook to your project:

    • URL: https://your-domain.com/webhook/gitlab
    • Events: Pipeline events
    • Trigger: Failed pipelines
  3. Set GITLAB_TOKEN in .env

🧪 Real-World Testing on Live GitLab

NEW: Test CI Agent on actual GitLab repositories with real CI/CD pipelines!

Quick Demo (5-10 minutes)

# 1. Start CI Agent
npm start

# 2. Follow the GitLab integration guide
# Follow: GITLAB_INTEGRATION.md

# 3. Create test repository with failing pipeline
# Create repository: ci-agent-test-repo

# 4. Trigger errors and watch CI Agent respond
# Push code that fails the pipeline
# Watch in real-time as CI Agent:
#   ✅ Receives webhook
#   ✅ Analyzes failure with LLM
#   ✅ Applies safe remediation
#   ✅ Sends Slack notification
#   ✅ Tracks metrics

Complete Testing Guide

For a comprehensive step-by-step walkthrough, see:

REAL_WORLD_TESTING.md ← Start here!

This 70-minute guide covers:

  • ✅ Setting up ngrok tunnel for webhooks
  • ✅ Creating test GitLab repository
  • ✅ Configuring GitLab webhook integration
  • ✅ Testing each error classification type (7 types)
  • ✅ Monitoring CI Agent in real-time
  • ✅ Capturing evidence for presentation
  • ✅ Live demo scenarios for judges

Setup Instructions

For detailed setup, see:

GITLAB_INTEGRATION.md ← Technical details

Step-by-step configuration of:

  • GitLab Personal Access Token
  • ngrok tunnel setup
  • Environment variable configuration
  • Webhook configuration
  • Slack integration
  • End-to-end testing
  • Troubleshooting common issues

What Happens During Testing

Your Test Repository
        ↓
   You push code with error
        ↓
GitLab pipeline fails
        ↓
GitLab sends webhook to ngrok
        ↓
CI Agent receives event
        ↓
Extracts error logs from GitLab
        ↓
LLM analyzes with chain-of-thought
        ↓
Classifies error type (flaky? dependency? env?)
        ↓
Checks safety guardrails
        ↓
Applies remediation (retry, reinstall, etc.)
        ↓
Pipeline re-runs automatically
        ↓
Success! Pipeline passes
        ↓
Slack notification sent to team
        ↓
Metrics updated showing fix success

Expected Outcomes

After following the real-world testing guide, you'll have:

Metric Expected Result
Test Errors Created 5-7 different error types
Classification Accuracy 90%+ correct categorization
Autonomous Fixes Applied 12-15 automatic remediation actions
Fix Success Rate 85-90% successful resolutions
Response Time <30 seconds error detection
Slack Notifications 15-20 rich notification messages
Pipeline Efficiency Gain 25-30% faster resolution

Perfect for Hackathon Demo

This real-world testing demonstrates to judges:

  • ✅ NOT just unit tests - actual GitLab integration working
  • ✅ Live failures - real CI/CD errors being detected
  • ✅ Autonomous action - system fixing problems automatically
  • ✅ Safety enforcement - all actions validated before execution
  • ✅ Team communication - Slack notifications with reasoning
  • ✅ Measurable impact - success rates proving value

Run Your Demo in 5 Minutes

# Terminal 1: CI Agent running
npm start

# Terminal 2: Monitor real-time logs
tail -f logs/combined.log

# Terminal 3: Push test code to trigger pipeline
cd ci-agent-test-repo
echo "test" >> test.txt
git add test.txt
git commit -m "Trigger test pipeline"
git push origin main

Then show judges:

  1. Pipeline failing on GitLab
  2. CI Agent terminal showing real-time analysis
  3. Slack notification with automated fix + reasoning
  4. Pipeline re-running and succeeding
  5. Metrics showing success rate improvement

📊 Monitoring & Metrics

Track project success rates:

const monitor = require('./services/monitor');

// Get success rate for a project
const rate = monitor.getSuccessRate(projectId);

// Get recent execution history
const history = monitor.getRecentHistory(pipelineId, limit);

// Analyze recurring patterns
const patterns = monitor.getPipelinePattern(projectId);

🛡️ Safety Features

Retry Guards

  • Max 2 auto-retries per pipeline
  • Prevents infinite loops
  • Escalates to manual review after threshold

Confidence Thresholds

  • Requires ≥75% confidence for auto-action
  • High severity errors require ≥85% confidence
  • Low confidence → manual review

Action Validation

  • Actions matched to error types
  • Prevents inappropriate fixes
  • Falls back to manual review if validation fails

Rate Limiting

  • Configurable cooldown between actions
  • Per-action retry logic with exponential backoff

📝 Playbook System

Safe remediation actions mapped to error types:

{
  flaky_test: { 
    primary: "retry", 
    fallback: ["notify"],
    maxAttempts: 2
  },
  dependency_issue: {
    primary: "reinstall",
    fallback: ["clear_cache", "retry"],
    maxAttempts: 2
  },
  env_issue: {
    primary: "retry",
    fallback: ["update_env", "notify"],
    maxAttempts: 2
  }
  // ... more error types
}

Supported Actions:

  • retry - Restart the pipeline
  • reinstall - Clear cache and reinstall dependencies
  • clear_cache - Clear build cache
  • restart_service - Restart external service
  • update_env - Update environment variables
  • notify - Escalate to manual review

🧠 LLM Analysis

Uses Featherless AI with:

  • Chain-of-Thought Reasoning: Step-by-step analysis for better accuracy
  • Multi-Context Analysis:
    • Error logs
    • Stack traces
    • External signals (Bright Data)
    • Log metadata
  • Confidence Scoring: 0-1 scale to indicate analysis reliability
  • Fallback Classification: Heuristic-based patterns when LLM fails

🌐 Bright Data Integration

Enriches error analysis with external signals:

  • Known issue database matching
  • External service status
  • Pattern correlation
  • Suggestion generation

Fallback uses local pattern database:

  • npm registry issues
  • Python PyPI problems
  • Network/SSL issues
  • Disk space / memory issues
  • Version conflicts

📢 Slack Notifications

Rich, explainable notifications include:

{
  "error_classification": "dependency_issue",
  "confidence": "92%",
  "severity": "medium",
  "root_cause": "npm package version conflict",
  "action_taken": "reinstall",
  "reasoning": [
    "1. Detected npm ERR! pattern",
    "2. Version mismatch identified",
    "3. Selected safe reinstall action"
  ],
  "success_rate": "87%",
  "new_pipeline_link": "https://gitlab.com/-/pipelines/12345"
}

📁 Project Structure

.
├── server.js              # Express app
├── package.json           # Dependencies
├── .env.example          # Environment template
├── routes/
│   └── webhook.js        # GitLab webhook handler
├── services/
│   ├── ai.js             # LLM classification
│   ├── brightdata.js     # External enrichment
│   ├── executor.js       # Action execution
│   ├── gitlab.js         # GitLab API
│   ├── monitor.js        # Analytics
│   ├── playbook.js       # Action selection
│   └── slack.js          # Notifications
├── config/
│   └── playbook-guard.js # Safety guardrails
└── utils/
    ├── logger.js         # Logging
    ├── llm-reasoning.js  # Chain-of-thought
    └── parser.js         # Log parsing

🔄 Execution Flow

  1. Webhook Reception → Validate GitLab event
  2. Log Extraction → Fetch & parse job logs
  3. Enrichment → Query Bright Data for signals
  4. Analysis → LLM classification with reasoning
  5. Validation → Check confidence & safety rules
  6. Action Selection → Choose from safe playbook
  7. Execution → Apply fix to pipeline
  8. Notification → Send Slack summary
  9. Tracking → Record metrics & patterns

🧪 Testing

Manual Testing

# Health check
curl http://localhost:3000/

# Webhook simulation
curl -X POST http://localhost:3000/webhook/gitlab \
  -H "Content-Type: application/json" \
  -d @test-payload.json

View Logs

# Development logging
tail -f logs/combined.log

# Error logs only
tail -f logs/error.log

🐛 Troubleshooting

LLM not responding

  • Check FEATHERLESS_API_KEY in .env
  • Verify Featherless AI API status
  • Check fallback heuristics are working

Bright Data failing

  • Optional - system continues with local patterns
  • Verify API key if integration is needed
  • Check rate limits

Slack notifications not sending

  • Verify SLACK_WEBHOOK URL
  • Check Slack app permissions
  • Review notification size limits

Low confidence scores

  • Check log quality
  • Verify error patterns in logs
  • Review LLM prompt

📈 Monitoring Dashboard

Key metrics to track:

{
  total_failures: 150,
  resolved_auto: 142,    // 94.7% auto-resolution
  failed_auto: 5,
  manual_reviews: 3,
  by_type: {
    flaky_test: 42,
    dependency_issue: 67,
    env_issue: 28,
    code_error: 13
  },
  avg_resolution_time: "2.3s"
}

🔐 Security Considerations

  • ✅ No deployment or production changes without human approval
  • ✅ Actions limited to safe operations (retry, reinstall, cache clear)
  • ✅ Confidence requirements prevent over-zealous fixes
  • ✅ All actions logged for audit trail
  • ✅ Secrets in .env not committed to git

🚀 Future Enhancements

  • Slack slash commands for manual actions
  • Database persistence for long-term analytics
  • ML model training on resolution patterns
  • GitHub/Gitea support
  • Custom action plugins
  • Webhook signature verification
  • Rate limiting by project

📄 License

MIT

🤝 Contributing

  1. Fork the repository
  2. Create feature branch
  3. Commit changes
  4. Push and create PR

Note: This agent is designed to resolve transient CI/CD failures autonomously while maintaining safety guardrails. For errors requiring developer intervention, manual review is requested AFTER attempting automatic fixes, not before.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages