Percepta

An AI-powered macOS automation agent that can perform computer tasks by analyzing screenshots and executing actions. Built with TypeScript and Claude AI.

Features

• 🤖 AI-Powered Automation: Uses Claude AI to understand screenshots and decide actions
• 🖥️ macOS Integration: Native screen capture and system automation
• 🎯 Precise Actions: Click, type, scroll, and navigate with pixel-perfect accuracy
• 🔄 Smart Recovery: Detects stuck patterns and applies recovery actions
• 📊 Performance Monitoring: Real-time metrics and optimization
• 🍞 Toast Notifications: Visual feedback during operation

Quick Start

Prerequisites

• macOS (tested on macOS 14+)
• Node.js 18+
• Anthropic API key

Installation

1. Clone the repository

   git clone https://github.com/KindredSM/percepta.git
   cd percepta

2. Install dependencies (includes building native components)

   npm install

3. Set up environment variables

   cp env.example .env
   # Edit .env and add your ANTHROPIC_API_KEY

4. Set up the agent command (optional, for convenience)

   # Make the agent script executable
   chmod +x agent
   
   # Add to your PATH (choose one method):
   # Method 1: Add to shell profile
   echo 'export PATH="'$(pwd)':$PATH"' >> ~/.zshrc
   source ~/.zshrc
   
   # Method 2: Create symlink (if you have write access)
   sudo ln -sf "$(pwd)/agent" /usr/local/bin/agent

Usage

Command Line Interface

# Run with a goal
npx tsx src/cli.ts "open calculator"

# Run with custom steps
npx tsx src/cli.ts "take a screenshot" --steps 10

# Using the agent command (after setup)
agent "open safari and go to google.com"

HTTP API

# Start the server
npm run dev

# Send a request
curl -X POST http://localhost:3030/agent/start \
  -H "Content-Type: application/json" \
  -d '{"goal": "open calculator", "steps": 5}'

Examples

# Open applications
agent "open calculator"
agent "launch safari"

# Web navigation
agent "go to github.com"
agent "search for 'typescript' on google"

# File operations
agent "create a new text file called hello.txt"
agent "take a screenshot"

# Form filling
agent "sign up for a new account on example.com"

Architecture

src/                          # TypeScript core
├── index.ts                  # HTTP server entry point
├── cli.ts                    # Command line interface
├── config.ts                 # Configuration management
├── orchestrator/
│   └── loop.ts              # Main agent loop logic
├── routes/
│   └── agent.ts             # HTTP API routes
├── services/
│   ├── actuator.ts          # Action execution
│   ├── screen.ts            # Screen capture
│   ├── visionProvider.ts    # AI integration
│   ├── toast.ts             # Notifications
│   ├── logger.ts            # Logging
│   └── performance.ts       # Metrics
└── types/
    └── action.ts            # Type definitions

native/                       # Swift UI components
├── ToastOverlay/            # Toast notification system
├── MenuBarAgent/            # Menu bar integration
├── Makefile                 # Build system
└── dev.sh                   # Development scripts

Configuration

The project uses optimized defaults and requires minimal configuration:

Required

• ANTHROPIC_API_KEY: Your Anthropic API key

Optional

• DEBUG_RESPONSES: Set to 1 to enable debug logging

For advanced customization, you can modify src/config.ts directly.

Development

Scripts

# Unified build system (recommended)
make                 # Build everything (TypeScript + Swift)
make setup           # Install dependencies (includes build)
make dev             # Start development server
make watch-native    # Watch Swift components for changes
make clean           # Clean all build artifacts

# npm scripts (alternative)
npm run dev          # Start development server
npm run build        # Build TypeScript + Swift components
npm run build:native # Build Swift components only
npm run lint         # Run ESLint
npm start            # Start production server

Project Structure

• TypeScript: Modern ES modules with strict typing
• Koa: Lightweight HTTP framework
• Zod: Runtime type validation
• Anthropic SDK: Claude AI integration
• Swift: Native macOS UI components (toast notifications, menu bar)

Native Components

The project includes Swift components for native macOS integration:

• ToastOverlay: Shows progress notifications during agent operation
• MenuBarAgent: Menu bar integration for quick access

Build automatically included when you run npm install.

MenuBarAgent Setup

The MenuBarAgent provides a menu bar icon for quick access to Percepta. After building:

1. Run the MenuBarAgent:

   cd native/MenuBarAgent && ./MenuBarAgent

2. Grant permissions (if prompted):

   • Go to System Preferences → Security & Privacy → Privacy → Accessibility
   • Add MenuBarAgent and allow it

3. Look for the menu bar icon (usually appears in the top-right menu bar)

Note: If the MenuBarAgent doesn't appear, check:

• macOS Accessibility permissions are granted
• Menu bar has space available
• No other MenuBarAgent processes are running (ps aux | grep MenuBarAgent)

For development:

make watch-native    # Watch Swift components for changes
make test-native     # Test Swift components

See native/README.md for detailed Swift development instructions.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Project Name

Percepta - From "perception" + "automation", representing the agent's ability to perceive and act on visual information.

Security

• API keys are stored in environment variables
• No sensitive data is logged
• Actions are sandboxed to user permissions

Troubleshooting

Common Issues

1. "ANTHROPIC_API_KEY not set"

   • Ensure your .env file contains the API key

2. Permission errors

   • Grant accessibility permissions to Terminal/your IDE
   • Enable screen recording permissions

3. Build errors

   • Ensure Node.js 18+ is installed
   • Run npm install to install dependencies

Debug Mode

Set DEBUG_RESPONSES=1 to see raw AI responses:

DEBUG_RESPONSES=1 agent "your goal here"

Roadmap