Skip to content

Latest commit

 

History

History

example

Folders and files

README.md

Cluster Example - HTTP API with Raft Cluster

This example demonstrates how to build a distributed key-value store with HTTP API using rockraft in cluster mode.

Overview

This example shows:

  • Creating a multi-node Raft cluster (3 nodes)
  • Cluster mode with join configuration
  • HTTP API for read/write operations
  • Leader election and automatic failover
  • Data replication across all nodes

Architecture

                    HTTP Clients
                        |
                        v
+-----------------+  +-----------------+  +-----------------+
|   Node 1        |  |   Node 2        |  |   Node 3        |
| (HTTP: 8001)    |  | (HTTP: 8002)    |  | (HTTP: 8003)    |
| (Raft: 7001)    |  | (Raft: 7002)    |  | (Raft: 7003)    |
|                 |  |                 |  |                 |
| RocksDB         |  | RocksDB         |  | RocksDB         |
|                 |  |                 |  |                 |
+--------+--------+  +--------+--------+  +--------+--------+
         |                      |                      |
         +---------- Raft Consensus (gRPC) ---------+

Project Structure

example/
├── Cargo.toml              # Project dependencies
├── start.sh                # Cluster management script
├── conf/                   # Configuration files
│   ├── node1.toml         # Node 1 configuration
│   ├── node2.toml         # Node 2 configuration
│   └── node3.toml         # Node 3 configuration
├── src/
│   └── main.rs            # HTTP server with Raft integration
├── logs/                   # Log directory (created by start.sh)
└── README.md              # This file

HTTP API Endpoints

GET /get?key={key}

Get a value by key.

Example:

curl "http://localhost:8001/get?key=mykey"

Response:

{
  "key": "mykey",
  "value": "myvalue"
}

If key doesn't exist:

{
  "key": "mykey",
  "value": null
}

POST /set

Set a key-value pair (only works on leader node).

Example:

curl -X POST http://localhost:8001/set \
  -H "Content-Type: application/json" \
  -d '{"key": "mykey", "value": "myvalue"}'

Response (Success):

{
  "success": true,
  "message": "Key 'mykey' set successfully"
}

Response (Not Leader):

{
  "error": "This node is not the leader. Current leader: 1"
}

POST /delete

Delete a key (only works on leader node).

Example:

curl -X POST http://localhost:8001/delete \
  -H "Content-Type: application/json" \
  -d '{"key": "mykey"}'

Response:

{
  "success": true,
  "message": "Key 'mykey' deleted successfully"
}

POST /batch_write

Batch write multiple keys atomically (only works on leader node).

Example:

curl -X POST http://localhost:8001/batch_write \
  -H "Content-Type: application/json" \
  -d '{
    "operations": [
      { "op": "set", "key": "key1", "value": "value1" },
      { "op": "set", "key": "key2", "value": "value2" },
      { "op": "delete", "key": "key3" }
    ]
  }'

Response:

{
  "success": true,
  "message": "Batch write successful: 3 operations applied"
}

POST /txn

Execute a transaction with conditional operations (only works on leader node). All conditions must be met (AND logic) for if_then operations to execute. Otherwise, else_then operations are executed.

Condition operators: exists, not_exists, equal, not_equal, greater, less, greater_equal, less_equal

Example (CAS - Compare and Swap):

curl -X POST http://localhost:8001/txn \
  -H "Content-Type: application/json" \
  -d '{
    "conditions": [
      { "key": "counter", "op": "exists" },
      { "key": "counter", "op": "equal", "value": "10" }
    ],
    "if_then": [
      { "op": "set", "key": "counter", "value": "20" }
    ],
    "else_then": [
      { "op": "set", "key": "counter", "value": "0" }
    ],
    "return_previous": true
  }'

Response (conditions met):

{
  "success": true,
  "branch": true,
  "prev_values": ["10"]
}

Response (conditions not met):

{
  "success": true,
  "branch": false,
  "prev_values": []
}

POST /getset

Atomically get the old value and set a new value (only works on leader node). Returns the previous value of the key, or null if the key did not exist.

Example:

curl -X POST http://localhost:8001/getset \
  -H "Content-Type: application/json" \
  -d '{"key": "mykey", "value": "new_value"}'

Response (key existed):

{
  "success": true,
  "key": "mykey",
  "old_value": "old_value",
  "new_value": "new_value"
}

Response (key did not exist):

{
  "success": true,
  "key": "mykey",
  "old_value": null,
  "new_value": "new_value"
}

POST /join

Add a node to the cluster (only works on leader node, forwarded automatically).

Example:

curl -X POST http://localhost:8001/join \
  -H "Content-Type: application/json" \
  -d '{"node_id": 4, "endpoint": "127.0.0.1:7004"}'

Response:

{
  "success": true,
  "message": "Node 4 joined the cluster successfully"
}

POST /leave

Remove a node from the cluster (only works on leader node, forwarded automatically).

Example:

curl -X POST http://localhost:8001/leave \
  -H "Content-Type: application/json" \
  -d '{"node_id": 3}'

Response:

{
  "success": true,
  "message": "Node 3 left the cluster successfully"
}

GET /members

Get the cluster membership as the leader has applied it. Followers forward the request, so every node answers with the same view.

Example:

curl http://localhost:8001/members

Response:

{
  "success": true,
  "voters": ["1", "2", "3"],
  "learners": [],
  "members": {
    "1": {"node_id": 1, "endpoint": "127.0.0.1:7001", "voter": true},
    "2": {"node_id": 2, "endpoint": "127.0.0.1:7002", "voter": true},
    "3": {"node_id": 3, "endpoint": "127.0.0.1:7003", "voter": true}
  }
}

members holds every member with its endpoint, and voters / learners split that same set: a member missing from voters is a learner, which receives replication but does not vote. The two lists never overlap and always cover members exactly.

GET /health

Health check endpoint.

Example:

curl http://localhost:8001/health

Response:

{
  "status": "ok",
  "node_id": 1,
  "is_leader": true,
  "current_leader": 1,
  "current_term": 1,
  "state": "Leader"
}

GET /metrics

Get detailed cluster metrics.

Example:

curl http://localhost:8001/metrics

Response:

{
  "node_id": 1,
  "current_leader": 1,
  "current_term": 1,
  "last_applied": { ... },
  "last_log_index": 2,
  "state": "Leader",
  "membership": { ... }
}

Cluster Management Script

The start.sh script provides an easy way to manage the Raft cluster.

Script Features

  • One-command startup: Start all three nodes with ./start.sh start
  • Status monitoring: Check which nodes are running with ./start.sh status
  • Log viewing: View real-time logs for any node with ./start.sh logs <node>
  • Cluster testing: Automated test script to verify cluster functionality
  • Graceful shutdown: Stop all nodes with proper cleanup
  • Automatic cleanup: Clean up log files when needed

Script Commands

Command Description
./start.sh start Start all three cluster nodes sequentially
./start.sh stop Stop all running cluster nodes
./start.sh restart Stop and restart all nodes
./start.sh status Show status of all nodes (running/stopped)
./start.sh logs <node> View real-time logs for a specific node
./start.sh test Test cluster health and data replication
./start.sh clean Remove all log files in logs/ directory

Usage Examples

# Build the project first
cargo build

# Start the cluster
./start.sh start

# Check status after a few seconds
./start.sh status

# View logs from node 1
./start.sh logs node1

# Test the cluster (writes a key and reads from all nodes)
./start.sh test

# Stop all nodes when done
./start.sh stop

Log Files

The script creates a logs/ directory containing:

logs/
├── node1.log       # Node 1 output
├── node1.pid       # Node 1 process ID
├── node2.log       # Node 2 output
├── node2.pid       # Node 2 process ID
├── node3.log       # Node 3 output
└── node3.pid       # Node 3 process ID

Configuration Files

Each node has its own TOML configuration file in the conf/ directory.

Node 1 (conf/node1.toml)

node_id = 1
http_address = "127.0.0.1:8001"

[raft]
address = "127.0.0.1:7001"
join = ["127.0.0.1:7002", "127.0.0.1:7003"]

[rocksdb]
data_path = "/tmp/rockraft_cluster/node1"
max_open_files = 10000

Node 2 (conf/node2.toml)

node_id = 2
http_address = "127.0.0.1:8002"

[raft]
address = "127.0.0.1:7002"
join = ["127.0.0.1:7001"]

[rocksdb]
data_path = "/tmp/rockraft_cluster/node2"
max_open_files = 10000

Node 3 (conf/node3.toml)

node_id = 3
http_address = "127.0.0.1:8003"

[raft]
address = "127.0.0.1:7003"
join = ["127.0.0.1:7001"]

[rocksdb]
data_path = "/tmp/rockraft_cluster/node3"
max_open_files = 10000

How to Run

Step 1: Build the example

From the examples directory:

cargo build

Step 2: Start the cluster

Option A: Using the start script (recommended)

The start.sh script provides an easy way to manage the cluster:

# Start all three nodes
./start.sh start

# Check cluster status
./start.sh status

# View logs for a specific node
./start.sh logs node1

# Test the cluster
./start.sh test

# Stop all nodes
./start.sh stop

# Restart all nodes
./start.sh restart

# Clean up log files
./start.sh clean

Option B: Manually starting each node

Open three separate terminals and start the nodes:

Terminal 1 - Node 1:

cargo run -- --conf conf/node1.toml

Terminal 2 - Node 2:

cargo run -- --conf conf/node2.toml

Terminal 3 - Node 3:

cargo run -- --conf conf/node3.toml

Step 3: Test the cluster

Check health of all nodes:

curl http://localhost:8001/health
curl http://localhost:8002/health
curl http://localhost:8003/health

Set a key-value pair (write to leader):

curl -X POST http://localhost:8001/set \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "Alice"}'

Get the value (read from any node):

curl "http://localhost:8001/get?key=user:1"
curl "http://localhost:8002/get?key=user:1"
curl "http://localhost:8003/get?key=user:1"

Set more data:

curl -X POST http://localhost:8001/set \
  -H "Content-Type: application/json" \
  -d '{"key": "user:2", "value": "Bob"}'

curl -X POST http://localhost:8001/set \
  -H "Content-Type: application/json" \
  -d '{"key": "config:theme", "value": "dark"}'

Delete a key:

curl -X POST http://localhost:8001/delete \
  -H "Content-Type: application/json" \
  -d '{"key": "config:theme"}'

Step 4: Stop the nodes

Press Ctrl+C in each terminal to gracefully shut down the nodes.

Example Output

Node 1 Startup Output

Configuration loaded:
  node_id: 1
  raft_addr: 127.0.0.1:7001
  http_addr: 127.0.0.1:8001
  data_path: /tmp/rockraft_cluster/node1
  join: ["127.0.0.1:7002", "127.0.0.1:7003"]

Creating Raft node...
✓ Raft node created successfully!

Initializing 3-node cluster...
✓ Cluster initialized successfully!

Waiting for initial log to be committed...
✓ Initial log committed successfully!

HTTP server listening on http://127.0.0.1:8001

Available endpoints:
  GET  http://127.0.0.1:8001/get?key=<key>    - Get a value by key
  POST http://127.0.0.1:8001/set             - Set a key-value pair
  POST http://127.0.0.1:8001/delete          - Delete a key
  GET  http://127.0.0.1:8001/health          - Health check
  GET  http://127.0.0.1:8001/metrics         - Cluster metrics

Press Ctrl+C to shutdown...

Node 2 Startup Output

Configuration loaded:
  node_id: 2
  raft_addr: 127.0.0.1:7002
  http_addr: 127.0.0.1:8002
  data_path: /tmp/rockraft_cluster/node2
  join: ["127.0.0.1:7001"]

Creating Raft node...
✓ Raft node created successfully!

Waiting for cluster to be ready...
Waiting for initial log to be committed...
✓ Initial log committed successfully!

HTTP server listening on http://127.0.0.1:8002

Available endpoints:
  GET  http://127.0.0.1:8002/get?key=<key>    - Get a value by key
  POST http://127.0.0.1:8002/set             - Set a key-value pair
  POST http://127.0.0.1:8002/delete          - Delete a key
  GET  http://127.0.0.1:8002/health          - Health check
  GET  http://127.0.0.1:8002/metrics         - Cluster metrics

Press Ctrl+C to shutdown...

Key Concepts

Cluster Mode

  • All nodes are configured with join list
  • Node 1 initializes the cluster with all 3 members
  • Node 2 and 3 join the cluster via the join configuration
  • Data is automatically replicated across all nodes

Leader Election

  • Node 1 becomes the initial leader by initializing the cluster
  • If Node 1 fails, Nodes 2 and 3 will elect a new leader
  • Only the leader accepts write requests (POST /set, POST /delete)
  • All nodes accept read requests (GET /get); followers forward them to the leader

Data Consistency

  • Write operations go through Raft consensus
  • All nodes maintain identical data through log replication
  • Reads return locally stored data (eventually consistent)

Fault Tolerance

  • The cluster can tolerate 1 node failure (3 nodes total)
  • If leader fails, followers elect a new leader automatically
  • Data is safely replicated across all nodes

Implementation Details

Write Path (POST /set)

  1. Client sends POST request to any node
  2. Node checks if it's the leader via assume_leader()
  3. If leader: writes to Raft via client_write()
  4. If not leader: returns error with leader information
  5. Raft replicates the write to all followers
  6. Write is committed once majority acknowledges
  7. State machine applies the write to RocksDB

Read Path (GET /get)

  1. Client sends GET request to any node
  2. If the node is not the leader, the read is forwarded to the leader
  3. The leader reads from its applied state machine (RocksDB)
  4. Returns value or null if key doesn't exist

Note: reads are not linearizable — they return the leader's last-applied state, which may briefly lag the latest commit. Routing through the leader keeps a single consistent read point while avoiding cross-node replication coordination per read.

Next Steps

To build a production-ready Raft cluster:

  1. Implement TLS: Add HTTPS support for secure communication
  2. Authentication: Add JWT or token-based authentication
  3. Load Balancing: Use a load balancer to distribute HTTP requests
  4. Monitoring: Add Prometheus metrics and Grafana dashboards
  5. Backup: Implement automated backup of RocksDB data
  6. Cluster Management: Add API endpoints for adding/removing nodes
  7. Read Consistency: Implement linearizable reads if needed
  8. Rate Limiting: Add rate limiting to prevent abuse

Dependencies

  • rockraft: The main Raft library
  • axum: HTTP web framework
  • tower-http: HTTP middleware (CORS, tracing)
  • tokio: Async runtime
  • rocksdb: Embedded database
  • serde & serde_json: Serialization
  • toml: Configuration file parsing
  • tracing: Structured logging

Troubleshooting

Port Already in Use

If you see "port already in use" error:

  • Stop all running nodes with Ctrl+C
  • Check if ports are still in use: lsof -i :7001 -i :8001
  • Restart the nodes

Nodes Cannot Connect

If nodes cannot join the cluster:

  • Ensure nodes are started in the correct order (node1 first)
  • Check that ports 7001-7003 are not blocked by firewall
  • Verify network connectivity between nodes
  • Check the join addresses in configuration files

Write Operations Fail

If POST /set fails with "not leader" error:

  • The node you're writing to is not the leader
  • Check /health endpoint to find the current leader
  • Direct writes to the leader node's HTTP port
  • Or implement client-side leader redirection

Data Not Persisting

If data is not persisting:

  • Check that /tmp/rockraft_cluster directory exists and is writable
  • Ensure RocksDB has sufficient disk space
  • Verify file permissions
  • Check logs for errors

Notes

  • Data is stored in /tmp/rockraft_cluster/node{N} directories
  • Raft communication uses ports 7001-7003 (gRPC)
  • HTTP API uses ports 8001-8003
  • All nodes communicate via gRPC for Raft consensus
  • The cluster requires at least 2 nodes to maintain a quorum
  • Node 1 initializes the cluster, others join via configuration
  • Use start.sh script for easy cluster management:
    • Run ./start.sh start to launch all nodes
    • Run ./start.sh status to check cluster health
    • Run ./start.sh logs <node> to view node logs
    • Run ./start.sh stop to gracefully shut down all nodes
  • If starting nodes manually, press Ctrl+C in each terminal to shut down