ACTIVE DEVELOPMENT: Redstone is currently a distributed cache, it does not have support for replication yet. Fault tolerance and replication are actively being worked upon.
A high-performance, distributed tensor cache built in Rust, optimized for ML inference and training workloads.
Complete:
- Single-node LRU cache with memory-based eviction
- streaming gRPC
- Connection pooling for improved parallelism
- Network protocol (Protocol Buffers)
- Basic put/get/delete/observability operations
- Consistent hashing for sharding
- Multi-node cluster support
- Client side caching
- Basic Control plane for cluster control
Planned:
- Compression
- Memory tiering
- Zero copy serialization/deserialization
- Metrics for observability into cache ops
- Better replication and data availability
Known Limitations:
- Extremely simple design (will not scale for very large tensors)
- No replication
- Data loss in case of server death
- streaming gRPC for all tensor sizes
- SerDe ops need copies
- no client
- Rust 1.70+
- Protocol Buffers compiler (for building from source)
# Install protoc (if needed)
# macOS
brew install protoc
# Ubuntu/Debian
apt-get install protobuf-compilergit clone https://github.com/pri1712/redstone.git
cd redstone
cargo build --releaseStart a cache server with 1GB of memory:
cargo run --release --bin redstone_server
# Server starts on 127.0.0.1:50051 with 1GB cacheOr customize the address and cache size:
cargo run --release --bin redstone_server 0.0.0.0:50051 10737418240
# 10GB cache on all interfacesTry the example client to test the server:
# In another terminal
cargo run --bin redstone_client
# The client should be able to connect to the server which was started in the previous step.- Memory-efficient LRU eviction: Evicts based on tensor size, not count
- High performance: O(1) get/put operations
- Thread-safe: Concurrent access from multiple clients
- Network protocol: gRPC for client-server communication
- Type-safe: Protocol Buffers for wire format
- Multiple data types: F32, F64, I32, I64, U8
- Flexible layouts: Row-major and column-major storage
- Distributed sharding: Consistent hashing across N nodes
- Replication: Configurable replication factor (default: 3)
- Fault tolerance: Survive node failures
- Client-side caching: L1 cache for hot data
- Compression: LZ4 for large tensors
- Monitoring: Prometheus metrics
- Multi-language clients: Python, Go, Java, JavaScript
- Tiered data storage: Tiering for data, to allow for better memory management and durability.
graph TD
subgraph ClientSide ["Client Side"]
direction TB
S1[Service 1]
S2[Service 2]
S3[Service 3]
RC[Redstone client library]
S1 & S2 & S3 --> RC
end
subgraph ServerSide ["Server Side"]
direction LR
RS1[Redstone server]
RS2[Redstone server]
RS3[Redstone server]
end
RC -- "gRPC" --> RS1
RC -- "gRPC" --> RS2
RC -- "gRPC" --> RS3
%% Annotation for the hashing strategy
subgraph Strategy [ ]
direction TB
style Strategy fill:none,stroke:none
Text[Consistent hashing strategy for<br/>distributing load across servers]
end
Text -.- RC
See ARCHITECTURE.md for detailed design decisions.
The server is a standalone binary that runs the cache service:
use redstone::client::RemoteCacheClient;
use redstone::tensor::meta::{TensorMeta, DType, StorageLayout};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Connect to cache server
let mut client = RemoteCacheClient::connect("127.0.0.1:50051".to_string()).await?;
// Create tensor metadata
let meta = TensorMeta::new(
DType::F32, // float32
vec![100, 768], // Shape: 100 x 768
StorageLayout::RowMajor,
)?;
// Prepare tensor data (76,800 floats = 307,200 bytes)
let data = vec![0.0f32; 100 * 768];
let bytes: Vec<u8> = unsafe {
std::slice::from_raw_parts(
data.as_ptr() as *const u8,
data.len() * 4,
).to_vec()
};
// Put tensor in cache
client.put("embeddings:user123".to_string(), meta, bytes).await?;
// Get tensor from cache
if let Some(tensor) = client.get("embeddings:user123").await? {
println!("Found tensor with shape: {:?}", tensor.meta().shape());
}
// Get cache statistics
let stats = client.get_stats().await?;
println!("Hit rate: {:.1}%", stats.hit_rate * 100.0);
println!("Memory used: {} MB", stats.memory_used / 1024 / 1024);
Ok(())
}The server exposes these RPC methods:
Retrieve a tensor by key.
Request:
message GetRequest {
string key = 1;
}Response:
message GetResponse {
TensorMeta meta = 1;
bytes data = 2;
}Errors:
NOT_FOUND: Key doesn't exist in cache
Store a tensor in the cache.
Request:
message PutRequest {
string key = 1;
TensorMeta meta = 2;
bytes data = 3;
}Response:
message PutResponse {} // Empty on successErrors:
AlreadyExists: Key already exists (immutable writes)OutOfMemory: Cache is full (eviction couldn't free enough space), this usually means that your tensor is larger than the configured cache size.InvalidTensor: Invalid tensorInvalidTensorMetadata: Invalid tensor metadataInvalidSize: Invalid tensor size
Remove a tensor from the cache.
Request:
message DeleteRequest {
string key = 1;
}Response:
message DeleteResponse {
bool deleted = 1; // true if key existed
}Get cache statistics.
Request:
message StatsRequest {}Response:
message StatsResponse {
uint64 entries = 1; // Number of cached tensors
uint64 memory_used = 2; // Bytes currently used
uint64 memory_limit = 3; // Max bytes allowed
uint64 hits = 4; // Cache hits
uint64 misses = 5; // Cache misses
uint64 evictions = 6; // LRU evictions
double hit_rate = 7; // hits / (hits + misses)
double memory_utilization = 8; // memory_used / memory_limit
}Cache expensive embedding computations:
async fn get_user_embedding(user_id: &str, cache: &mut RemoteCacheClient) -> Embedding {
let key = format!("embedding:user:{}", user_id);
// Try cache first
if let Ok(Some(tensor)) = cache.get(&key).await {
return tensor_to_embedding(tensor);
}
// Cache miss - compute with BERT (100ms)
let embedding = compute_bert_embedding(user_id).await;
// Store for next time
let _ = cache.put(key, embedding_to_tensor(&embedding)).await;
embedding
}Cache preprocessed training data:
async fn get_training_batch(image_ids: Vec<u64>, cache: &mut RemoteCacheClient) -> Vec<Tensor> {
let mut batch = Vec::new();
for id in image_ids {
let key = format!("preprocessed:{}", id);
if let Some(tensor) = cache.get(&key).await.unwrap() {
batch.push(tensor); // Cache hit - fast
} else {
// Cache miss - load and preprocess (slow)
let tensor = load_and_preprocess_image(id).await;
let _ = cache.put(key, tensor.clone()).await;
batch.push(tensor);
}
}
batch
}Share model weights across inference pods:
async fn load_model_weights(cache: &mut RemoteCacheClient) -> Model {
let layers = vec!["encoder.0.weight", "encoder.1.weight", /* ... */];
let mut weights = Vec::new();
for layer in layers {
if let Some(tensor) = cache.get(layer).await.unwrap() {
weights.push(tensor); // Shared across all pods
} else {
// Load from checkpoint (slow)
let weight = load_from_disk(layer).await;
cache.put(layer.to_string(), weight.clone()).await.unwrap();
weights.push(weight);
}
}
Model::from_weights(weights)
}
// First pod: Loads from disk (10s)
// Other pods: Load from cache (100ms) - 100x fasterNote: Benchmarks are for single-node setup, are reported without any network overhead (pure operative costs) and are performed on commercial hardware
- CPU: Apple M4 Chip (10 core)
- RAM: 16GB
- OS: OSX
- Rust version: 1.92.0
| Tensor Size | Average Latency | Throughput |
|---|---|---|
| 3KB (tiny - BERT embedding) | 0.4 μs | ~2.5M ops/sec |
| 50KB (small - layer weights) | 0.9 μs | ~1.1M ops/sec |
| 1MB (medium - weight matrix) | 6.2 μs | ~161K ops/sec |
| 10MB (large - full layer) | 20.8 μs | ~48K ops/sec |
| Tensor Size | p50 Latency | p99 Latency | Notes |
|---|---|---|---|
| 3KB (tiny) | 12.1 ns | 14.8 ns | BERT embedding lookup |
| 1MB (medium) | 12.4 ns | 16.2 ns | Arc::clone overhead only |
Throughput: ~56.8M reads/sec (single-threaded, cached data)
The above data ignores all network overhead, serialization and deserialization overhead.
Max sustained read ops: 17.6 ns/op = ~56.8M operations/sec
| Metric | Value |
|---|---|
| Mean | 17.609 ns |
| Median | 17.635 ns |
| Std Dev | 507 ps |
| R² | 0.932 |
https://docs.google.com/spreadsheets/d/1BdiEQra7bneoigugCkqQgkqMqycfD6QewtSLw30YVZQ/edit?usp=sharing For end to end benchmarks on the distributed architecture visit above link as of now.
Run benchmarks yourself:
cargo bench# Server
cargo run --bin server -- <address> <cache_size>
# Examples:
cargo run --bin server -- 0.0.0.0:50051 10737418240# Clone repository
git clone https://github.com/pri1712/redstone.git
cd redstone
# Build
cargo build --release
# Run tests
cargo test
# Run benchmarks
cargo bench
# Generate documentation
cargo doc --open# Unit tests
cargo test
# Integration tests (requires server running)
cargo test --test integration_test
# With logging
RUST_LOG=debug cargo testContributions are welcome! See CONTRIBUTING.md for guidelines.
Good first issues:
- Add more tensor dtypes (F16, BF16)
- Improve error messages
- Add metrics/logging
- Write integration tests
- Documentation improvements
A: Not yet. Currently single-node only. Distributed features (sharding, replication) are in active development.
A: LRU eviction automatically removes the least recently used tensors to make space. If a single tensor is larger than the entire cache, it will cause all other tensors to evict and then will fail the put operation.
A: No. Writes are immutable. To update, delete the old key and put a new value. This simplifies consistency in distributed mode.
A: Limited only by available memory and network transfer time. For very large tensors (>10GB), consider chunking.
A: No, currently in-memory only for maximum performance. Persistence is a potential future feature.
A: Yes! Convert your framework's tensors to bytes, then use the cache. Python client is planned for easier integration.
Apache License 2.0 - see LICENSE for details.
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Email: priyanshu17@outlook.com
Contributions welcome - see CONTRIBUTING.md