This example demonstrates how to build a distributed key-value store with HTTP API using rockraft in cluster mode.
This example shows:
- Creating a multi-node Raft cluster (3 nodes)
- Cluster mode with
joinconfiguration - HTTP API for read/write operations
- Leader election and automatic failover
- Data replication across all nodes
HTTP Clients
|
v
+-----------------+ +-----------------+ +-----------------+
| Node 1 | | Node 2 | | Node 3 |
| (HTTP: 8001) | | (HTTP: 8002) | | (HTTP: 8003) |
| (Raft: 7001) | | (Raft: 7002) | | (Raft: 7003) |
| | | | | |
| RocksDB | | RocksDB | | RocksDB |
| | | | | |
+--------+--------+ +--------+--------+ +--------+--------+
| | |
+---------- Raft Consensus (gRPC) ---------+
example/
├── Cargo.toml # Project dependencies
├── start.sh # Cluster management script
├── conf/ # Configuration files
│ ├── node1.toml # Node 1 configuration
│ ├── node2.toml # Node 2 configuration
│ └── node3.toml # Node 3 configuration
├── src/
│ └── main.rs # HTTP server with Raft integration
├── logs/ # Log directory (created by start.sh)
└── README.md # This file
Get a value by key.
Example:
curl "http://localhost:8001/get?key=mykey"Response:
{
"key": "mykey",
"value": "myvalue"
}If key doesn't exist:
{
"key": "mykey",
"value": null
}Set a key-value pair (only works on leader node).
Example:
curl -X POST http://localhost:8001/set \
-H "Content-Type: application/json" \
-d '{"key": "mykey", "value": "myvalue"}'Response (Success):
{
"success": true,
"message": "Key 'mykey' set successfully"
}Response (Not Leader):
{
"error": "This node is not the leader. Current leader: 1"
}Delete a key (only works on leader node).
Example:
curl -X POST http://localhost:8001/delete \
-H "Content-Type: application/json" \
-d '{"key": "mykey"}'Response:
{
"success": true,
"message": "Key 'mykey' deleted successfully"
}Batch write multiple keys atomically (only works on leader node).
Example:
curl -X POST http://localhost:8001/batch_write \
-H "Content-Type: application/json" \
-d '{
"operations": [
{ "op": "set", "key": "key1", "value": "value1" },
{ "op": "set", "key": "key2", "value": "value2" },
{ "op": "delete", "key": "key3" }
]
}'Response:
{
"success": true,
"message": "Batch write successful: 3 operations applied"
}Execute a transaction with conditional operations (only works on leader node).
All conditions must be met (AND logic) for if_then operations to execute.
Otherwise, else_then operations are executed.
Condition operators: exists, not_exists, equal, not_equal, greater, less, greater_equal, less_equal
Example (CAS - Compare and Swap):
curl -X POST http://localhost:8001/txn \
-H "Content-Type: application/json" \
-d '{
"conditions": [
{ "key": "counter", "op": "exists" },
{ "key": "counter", "op": "equal", "value": "10" }
],
"if_then": [
{ "op": "set", "key": "counter", "value": "20" }
],
"else_then": [
{ "op": "set", "key": "counter", "value": "0" }
],
"return_previous": true
}'Response (conditions met):
{
"success": true,
"branch": true,
"prev_values": ["10"]
}Response (conditions not met):
{
"success": true,
"branch": false,
"prev_values": []
}Atomically get the old value and set a new value (only works on leader node).
Returns the previous value of the key, or null if the key did not exist.
Example:
curl -X POST http://localhost:8001/getset \
-H "Content-Type: application/json" \
-d '{"key": "mykey", "value": "new_value"}'Response (key existed):
{
"success": true,
"key": "mykey",
"old_value": "old_value",
"new_value": "new_value"
}Response (key did not exist):
{
"success": true,
"key": "mykey",
"old_value": null,
"new_value": "new_value"
}Add a node to the cluster (only works on leader node, forwarded automatically).
Example:
curl -X POST http://localhost:8001/join \
-H "Content-Type: application/json" \
-d '{"node_id": 4, "endpoint": "127.0.0.1:7004"}'Response:
{
"success": true,
"message": "Node 4 joined the cluster successfully"
}Remove a node from the cluster (only works on leader node, forwarded automatically).
Example:
curl -X POST http://localhost:8001/leave \
-H "Content-Type: application/json" \
-d '{"node_id": 3}'Response:
{
"success": true,
"message": "Node 3 left the cluster successfully"
}Get the cluster membership as the leader has applied it. Followers forward the request, so every node answers with the same view.
Example:
curl http://localhost:8001/membersResponse:
{
"success": true,
"voters": ["1", "2", "3"],
"learners": [],
"members": {
"1": {"node_id": 1, "endpoint": "127.0.0.1:7001", "voter": true},
"2": {"node_id": 2, "endpoint": "127.0.0.1:7002", "voter": true},
"3": {"node_id": 3, "endpoint": "127.0.0.1:7003", "voter": true}
}
}members holds every member with its endpoint, and voters / learners split
that same set: a member missing from voters is a learner, which receives
replication but does not vote. The two lists never overlap and always cover
members exactly.
Health check endpoint.
Example:
curl http://localhost:8001/healthResponse:
{
"status": "ok",
"node_id": 1,
"is_leader": true,
"current_leader": 1,
"current_term": 1,
"state": "Leader"
}Get detailed cluster metrics.
Example:
curl http://localhost:8001/metricsResponse:
{
"node_id": 1,
"current_leader": 1,
"current_term": 1,
"last_applied": { ... },
"last_log_index": 2,
"state": "Leader",
"membership": { ... }
}The start.sh script provides an easy way to manage the Raft cluster.
- One-command startup: Start all three nodes with
./start.sh start - Status monitoring: Check which nodes are running with
./start.sh status - Log viewing: View real-time logs for any node with
./start.sh logs <node> - Cluster testing: Automated test script to verify cluster functionality
- Graceful shutdown: Stop all nodes with proper cleanup
- Automatic cleanup: Clean up log files when needed
| Command | Description |
|---|---|
./start.sh start |
Start all three cluster nodes sequentially |
./start.sh stop |
Stop all running cluster nodes |
./start.sh restart |
Stop and restart all nodes |
./start.sh status |
Show status of all nodes (running/stopped) |
./start.sh logs <node> |
View real-time logs for a specific node |
./start.sh test |
Test cluster health and data replication |
./start.sh clean |
Remove all log files in logs/ directory |
# Build the project first
cargo build
# Start the cluster
./start.sh start
# Check status after a few seconds
./start.sh status
# View logs from node 1
./start.sh logs node1
# Test the cluster (writes a key and reads from all nodes)
./start.sh test
# Stop all nodes when done
./start.sh stopThe script creates a logs/ directory containing:
logs/
├── node1.log # Node 1 output
├── node1.pid # Node 1 process ID
├── node2.log # Node 2 output
├── node2.pid # Node 2 process ID
├── node3.log # Node 3 output
└── node3.pid # Node 3 process ID
Each node has its own TOML configuration file in the conf/ directory.
node_id = 1
http_address = "127.0.0.1:8001"
[raft]
address = "127.0.0.1:7001"
join = ["127.0.0.1:7002", "127.0.0.1:7003"]
[rocksdb]
data_path = "/tmp/rockraft_cluster/node1"
max_open_files = 10000node_id = 2
http_address = "127.0.0.1:8002"
[raft]
address = "127.0.0.1:7002"
join = ["127.0.0.1:7001"]
[rocksdb]
data_path = "/tmp/rockraft_cluster/node2"
max_open_files = 10000node_id = 3
http_address = "127.0.0.1:8003"
[raft]
address = "127.0.0.1:7003"
join = ["127.0.0.1:7001"]
[rocksdb]
data_path = "/tmp/rockraft_cluster/node3"
max_open_files = 10000From the examples directory:
cargo buildOption A: Using the start script (recommended)
The start.sh script provides an easy way to manage the cluster:
# Start all three nodes
./start.sh start
# Check cluster status
./start.sh status
# View logs for a specific node
./start.sh logs node1
# Test the cluster
./start.sh test
# Stop all nodes
./start.sh stop
# Restart all nodes
./start.sh restart
# Clean up log files
./start.sh cleanOption B: Manually starting each node
Open three separate terminals and start the nodes:
Terminal 1 - Node 1:
cargo run -- --conf conf/node1.tomlTerminal 2 - Node 2:
cargo run -- --conf conf/node2.tomlTerminal 3 - Node 3:
cargo run -- --conf conf/node3.tomlcurl http://localhost:8001/health
curl http://localhost:8002/health
curl http://localhost:8003/healthcurl -X POST http://localhost:8001/set \
-H "Content-Type: application/json" \
-d '{"key": "user:1", "value": "Alice"}'curl "http://localhost:8001/get?key=user:1"
curl "http://localhost:8002/get?key=user:1"
curl "http://localhost:8003/get?key=user:1"curl -X POST http://localhost:8001/set \
-H "Content-Type: application/json" \
-d '{"key": "user:2", "value": "Bob"}'
curl -X POST http://localhost:8001/set \
-H "Content-Type: application/json" \
-d '{"key": "config:theme", "value": "dark"}'curl -X POST http://localhost:8001/delete \
-H "Content-Type: application/json" \
-d '{"key": "config:theme"}'Press Ctrl+C in each terminal to gracefully shut down the nodes.
Configuration loaded:
node_id: 1
raft_addr: 127.0.0.1:7001
http_addr: 127.0.0.1:8001
data_path: /tmp/rockraft_cluster/node1
join: ["127.0.0.1:7002", "127.0.0.1:7003"]
Creating Raft node...
✓ Raft node created successfully!
Initializing 3-node cluster...
✓ Cluster initialized successfully!
Waiting for initial log to be committed...
✓ Initial log committed successfully!
HTTP server listening on http://127.0.0.1:8001
Available endpoints:
GET http://127.0.0.1:8001/get?key=<key> - Get a value by key
POST http://127.0.0.1:8001/set - Set a key-value pair
POST http://127.0.0.1:8001/delete - Delete a key
GET http://127.0.0.1:8001/health - Health check
GET http://127.0.0.1:8001/metrics - Cluster metrics
Press Ctrl+C to shutdown...
Configuration loaded:
node_id: 2
raft_addr: 127.0.0.1:7002
http_addr: 127.0.0.1:8002
data_path: /tmp/rockraft_cluster/node2
join: ["127.0.0.1:7001"]
Creating Raft node...
✓ Raft node created successfully!
Waiting for cluster to be ready...
Waiting for initial log to be committed...
✓ Initial log committed successfully!
HTTP server listening on http://127.0.0.1:8002
Available endpoints:
GET http://127.0.0.1:8002/get?key=<key> - Get a value by key
POST http://127.0.0.1:8002/set - Set a key-value pair
POST http://127.0.0.1:8002/delete - Delete a key
GET http://127.0.0.1:8002/health - Health check
GET http://127.0.0.1:8002/metrics - Cluster metrics
Press Ctrl+C to shutdown...
- All nodes are configured with
joinlist - Node 1 initializes the cluster with all 3 members
- Node 2 and 3 join the cluster via the
joinconfiguration - Data is automatically replicated across all nodes
- Node 1 becomes the initial leader by initializing the cluster
- If Node 1 fails, Nodes 2 and 3 will elect a new leader
- Only the leader accepts write requests (
POST /set,POST /delete) - All nodes accept read requests (
GET /get); followers forward them to the leader
- Write operations go through Raft consensus
- All nodes maintain identical data through log replication
- Reads return locally stored data (eventually consistent)
- The cluster can tolerate 1 node failure (3 nodes total)
- If leader fails, followers elect a new leader automatically
- Data is safely replicated across all nodes
- Client sends POST request to any node
- Node checks if it's the leader via
assume_leader() - If leader: writes to Raft via
client_write() - If not leader: returns error with leader information
- Raft replicates the write to all followers
- Write is committed once majority acknowledges
- State machine applies the write to RocksDB
- Client sends GET request to any node
- If the node is not the leader, the read is forwarded to the leader
- The leader reads from its applied state machine (RocksDB)
- Returns value or null if key doesn't exist
Note: reads are not linearizable — they return the leader's last-applied state, which may briefly lag the latest commit. Routing through the leader keeps a single consistent read point while avoiding cross-node replication coordination per read.
To build a production-ready Raft cluster:
- Implement TLS: Add HTTPS support for secure communication
- Authentication: Add JWT or token-based authentication
- Load Balancing: Use a load balancer to distribute HTTP requests
- Monitoring: Add Prometheus metrics and Grafana dashboards
- Backup: Implement automated backup of RocksDB data
- Cluster Management: Add API endpoints for adding/removing nodes
- Read Consistency: Implement linearizable reads if needed
- Rate Limiting: Add rate limiting to prevent abuse
rockraft: The main Raft libraryaxum: HTTP web frameworktower-http: HTTP middleware (CORS, tracing)tokio: Async runtimerocksdb: Embedded databaseserde&serde_json: Serializationtoml: Configuration file parsingtracing: Structured logging
If you see "port already in use" error:
- Stop all running nodes with Ctrl+C
- Check if ports are still in use:
lsof -i :7001 -i :8001 - Restart the nodes
If nodes cannot join the cluster:
- Ensure nodes are started in the correct order (node1 first)
- Check that ports 7001-7003 are not blocked by firewall
- Verify network connectivity between nodes
- Check the
joinaddresses in configuration files
If POST /set fails with "not leader" error:
- The node you're writing to is not the leader
- Check
/healthendpoint to find the current leader - Direct writes to the leader node's HTTP port
- Or implement client-side leader redirection
If data is not persisting:
- Check that
/tmp/rockraft_clusterdirectory exists and is writable - Ensure RocksDB has sufficient disk space
- Verify file permissions
- Check logs for errors
- Data is stored in
/tmp/rockraft_cluster/node{N}directories - Raft communication uses ports 7001-7003 (gRPC)
- HTTP API uses ports 8001-8003
- All nodes communicate via gRPC for Raft consensus
- The cluster requires at least 2 nodes to maintain a quorum
- Node 1 initializes the cluster, others join via configuration
- Use
start.shscript for easy cluster management:- Run
./start.sh startto launch all nodes - Run
./start.sh statusto check cluster health - Run
./start.sh logs <node>to view node logs - Run
./start.sh stopto gracefully shut down all nodes
- Run
- If starting nodes manually, press Ctrl+C in each terminal to shut down