LogWatch - AI-Powered Log Analysis System

Overview

LogWatch is a production-ready, enterprise-grade log ingestion, search, and AI-powered analysis platform. It combines the power of OpenSearch for indexing, FastAPI for backend services, React for the frontend, and Groq's Llama 3 AI for intelligent log analysis.

Key Features

Core Capabilities

Real-time Log Ingestion: Automatically ingest logs from multiple sources with checkpoint-based incremental processing
Advanced Search: Full-text search with timestamp filtering, keyword search, and field-based queries
Interactive Dashboard: Modern React UI with time range selection, log visualization, and analytics
AI-Powered Analysis: Natural language chatbot for intelligent log analysis using Llama 3 70B model
Scalable Architecture: Distributed system design supporting millions of logs

AI Chatbot Features

Natural language queries (e.g., "Analyze errors from last hour")
Root cause analysis with severity classification
Actionable recommendations
Suggested OpenSearch queries for drill-down
Error timeline visualization
User feedback mechanism (thumbs up/down)

Technology Stack

Backend

Python 3.11: Core programming language
FastAPI: High-performance REST API framework
OpenSearch 2.x: Distributed search and analytics engine
Pydantic: Data validation and settings management
Groq API: LLM inference (Llama 3.3 70B)

Frontend

React 18: UI framework
Vite: Build tool and dev server
Recharts: Data visualization
TailwindCSS: Styling
Axios: HTTP client

Infrastructure

Docker Compose: Container orchestration
Nginx: Reverse proxy (optional)
SQLite: Checkpoint persistence

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    React Frontend (Port 3000)                │
│  ┌────────────────────────────────────────────────────┐     │
│  │  • Log Viewer      • Time Range Picker             │     │
│  │  • Search          • Analytics Charts              │     │
│  │  • AI Chat Bot     • Real-time Updates             │     │
│  └────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘
                            ↕ HTTP/REST API
┌─────────────────────────────────────────────────────────────┐
│                  FastAPI Backend (Port 8000)                 │
│  ┌────────────────────────────────────────────────────┐     │
│  │  API Endpoints:                                    │     │
│  │  • /api/logs       • /api/stats                    │     │
│  │  • /api/chat/analyze • /health                     │     │
│  └────────────────────────────────────────────────────┘     │
│  ┌────────────────────────────────────────────────────┐     │
│  │  Services:                                         │     │
│  │  • Log Parser      • AI Analyzer                   │     │
│  │  • Search Service  • Ingestion Worker              │     │
│  └────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────┘
                            ↕
┌─────────────────────────────────────────────────────────────┐
│              OpenSearch Cluster (Port 9200)                  │
│  • Index Management    • Full-text Search                    │
│  • Time-series Data    • Aggregations                        │
└─────────────────────────────────────────────────────────────┘
                            ↕
┌─────────────────────────────────────────────────────────────┐
│                    Groq API (Llama 3.3)                     │
│  • Log Analysis        • Root Cause Detection               │
│  • Recommendations     • Query Suggestions                  │
└─────────────────────────────────────────────────────────────┘

Installation

Prerequisites

Docker & Docker Compose
Python 3.11+
Node.js 18+
8GB+ RAM
Groq API key (free tier: https://console.groq.com)

Quick Start

Clone Repository

git clone <repository-url>
cd LogWatch

Configure Environment

# Backend configuration
cp .env.example .env

# Edit .env and add:
GROQ_API_KEY=your_groq_api_key_here
OPENSEARCH_PASSWORD=admin
REQUIRE_AUTH=false

Start Services

# Start all services
docker compose up -d

# Wait for OpenSearch to initialize (30 seconds)
sleep 30

# Setup OpenSearch indices
docker compose exec backend python scripts/setup_opensearch.py

Generate Sample Logs

# Generate realistic log scenarios
python scripts/generate_realistic_logs.py --output ./logs_in

# Ingest logs
docker compose exec backend python -m app.cli.ingest --directory /logs_in

Access Application

Frontend: http://localhost:3000
Backend API: http://localhost:8000
API Docs: http://localhost:8000/docs
OpenSearch: https://localhost:9200 (admin/admin)

Usage

Log Ingestion

Single File

python -m app.cli.ingest --file /path/to/logfile.log

Directory (Incremental)

python -m app.cli.ingest --directory /var/logs --incremental

Supported Formats

Plain text logs
JSON logs (one per line)
CSV logs
Apache/Nginx access logs

API Examples

Search Logs

curl "http://localhost:8000/api/logs?start_time=2025-10-24T00:00:00Z&end_time=2025-10-24T23:59:59Z&page_size=50"

Get Statistics

curl "http://localhost:8000/api/stats"

AI Analysis

curl -X POST "http://localhost:8000/api/chat/analyze" \
  -H "Content-Type: application/json" \
  -d '{
    "keywords": "ERROR",
    "time_window_minutes": 60
<<<<<<< Updated upstream
=======
  }' | jq '.summary'


-------------------------------------------------------------------------
debug Calls made
-------------------------------------------------------------------------

echo "=== Checking Log Timestamps ==="
curl -s -k -u admin:admin 'https://localhost:9200/logs-*/_search?pretty' \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 0,
    "aggs": {
      "min": {"min": {"field": "timestamp"}},
      "max": {"max": {"field": "timestamp"}}
    }
  }' | grep value_as_string

echo -e "\n=== Testing Chatbot Query ==="
curl -s -X POST http://localhost:8000/api/chat/analyze \
  -H "Content-Type: application/json" \
  -d '{
    "natural_language_query": "Analyze logs from last 5 hour"
  }' | jq '{
    keywords,
    time_window_minutes,
    time_range: .summary.time_range,
    total_logs: .summary.total_logs
>>>>>>> Stashed changes
  }'

AI Chatbot Usage

<<<<<<< Updated upstream In the UI, click the chat button and try:

"Analyze logs from last 30 minutes"
"Find all database errors"
"What's causing the payment failures?"
"Show me memory leak issues"

Configuration

Environment Variables

Variable	Default	Description
`OPENSEARCH_HOST`	opensearch-node1	OpenSearch hostname
`OPENSEARCH_PORT`	9200	OpenSearch port
`OPENSEARCH_USER`	admin	OpenSearch username
`OPENSEARCH_PASSWORD`	admin	OpenSearch password
`GROQ_API_KEY`	-	Groq API key for AI features
`GROQ_MODEL`	llama-3.3-70b-versatile	AI model name
`REQUIRE_AUTH`	false	Enable JWT authentication
`MAX_LOGS_PER_ANALYSIS`	200	Max logs sent to AI

OpenSearch Configuration

Index template settings in backend/app/search/mappings.py:

Number of shards: 1
Number of replicas: 0
Refresh interval: 5s
Codec: best_compression

Project Structure

LogWatch/
├── backend/
│   ├── app/
│   │   ├── ai/                  # AI analysis module
│   │   │   ├── analyzer.py      # Log analyzer
│   │   │   ├── providers.py     # AI provider abstraction
│   │   │   └── config.py        # AI settings
│   │   ├── api/
│   │   │   ├── routes.py        # REST endpoints
│   │   │   └── chat_routes.py   # AI chat endpoints
│   │   ├── auth/                # Authentication
│   │   ├── ingestion/           # Log ingestion
│   │   │   ├── worker.py        # Ingestion worker
│   │   │   └── parser.py        # Log parser
│   │   ├── search/
│   │   │   ├── client.py        # OpenSearch client
│   │   │   └── mappings.py      # Index templates
│   │   ├── config.py            # App configuration
│   │   └── main.py              # FastAPI app
│   ├── scripts/
│   │   ├── setup_opensearch.py
│   │   └── generate_realistic_logs.py
│   ├── tests/
│   ├── requirements.txt
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── components/
│   │   │   ├── Dashboard.jsx
│   │   │   ├── LogViewer.jsx
│   │   │   ├── ChatSidebar.jsx   # AI chatbot UI
│   │   │   └── ...
│   │   ├── services/
│   │   │   └── api.js
│   │   └── App.jsx
│   ├── package.json
│   └── Dockerfile
├── docker-compose.yml
├── .env
└── README.md

Testing

Backend Tests

cd backend
PYTHONPATH=backend pytest -v

Frontend Tests

cd frontend
npm test

Integration Tests

# Test complete workflow
./scripts/integration-test.sh

Performance

Benchmarks

Ingestion: 10,000 logs/second
Search: < 100ms for 1M logs
AI Analysis: 2-5 seconds for 200 logs
UI Response: < 500ms

Scalability

Tested with: 10M+ logs
Storage: ~1KB per log (compressed)
Memory: ~4GB for 1M logs
CPU: Minimal (< 10% on modern hardware)

Troubleshooting

No Logs Showing in UI

Check time range matches log timestamps
Verify backend connection: curl http://localhost:8000/health
Check OpenSearch: curl -k -u admin:admin 'https://localhost:9200/logs-*/_count?pretty'

AI Chat Not Working

Verify Groq API key in .env
Check model name: llama-3.3-70b-versatile
View backend logs: docker compose logs backend

OpenSearch Connection Failed

Check OpenSearch status: docker compose ps
Verify credentials in .env
Restart: docker compose restart opensearch-node1

Development

Adding New AI Providers

Create provider in backend/app/ai/providers.py
Implement AIProvider interface
Update get_ai_provider() factory

Custom Log Parsers

Edit backend/app/ingestion/parser.py
Add pattern to LOG_PATTERNS
Test with sample logs

Frontend Customization

Edit components in frontend/src/components/
Update API client in frontend/src/services/api.js
Build: npm run build

Deployment

Production Checklist

Docker Production Build

docker compose -f docker-compose.prod.yml up -d

Security

JWT-based authentication (optional)
OpenSearch TLS encryption
API key management for Groq
CORS configuration
Input validation with Pydantic

Roadmap

Real-time log streaming (WebSockets)
Alert system for anomaly detection
Multi-tenant support
Advanced analytics dashboard
Log correlation across services
Export reports (PDF/Excel)
Machine learning for anomaly detection

Contributing

Fork the repository
Create feature branch (git checkout -b feature/amazing-feature)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open Pull Request

License

MIT License - See LICENSE file for details

Support

Documentation: /docs
Issues: GitHub Issues
Email: support@logwatch.io

Acknowledgments

OpenSearch for search capabilities
Groq for AI inference
FastAPI for backend framework
React community for frontend tools

Built with ❤️ by Balaji PS

Version: 1.0.0 Created: September 2025 Last Updated: November 2025

Name		Name	Last commit message	Last commit date
backend		backend
docs		docs
frontend		frontend
logs_in		logs_in
scripts		scripts
.env		.env
.gitattributes		.gitattributes
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
debug_chatbot.sh		debug_chatbot.sh
docker-compose.yml		docker-compose.yml
env.example		env.example
package-lock.json		package-lock.json

Latest commit

History

Folders and files

Repository files navigation