ZetaDelta (ζΔ) Compression Engine (Rust Edition)

ZetaDelta is an enterprise-grade, high-performance, multi-column time-series database compression and delta-optimization engine written in native, hardware-optimized Rust.

By combining monotonic integer double-delta bit-packing (for timestamps), Gorilla-style floating-point XOR compression (for floats), and an AI-inspired Word-Level Vocabulary Tokenizer (for unstructured text), ZetaDelta consistently crushes standard general-purpose compressors (like GZIP and BZIP2) by 20% to 50% in size while offering 8x to 250x faster random-access query seeks.

---

✨ Features

• Auto-Schema Detection: Pass any tabular CSV file. The engine dynamically scans the headers and data values at runtime to automatically identify timestamps, float metrics, and categorical tags. No manual column indexing needed!
• Go & Python Compatibility: Our Rust binary formats are 100% synchronized and compatible with our Go and Python implementations.
• C-Compatible FFI Layer (zetadelta.dll): Compiles directly into a shared dynamic link library, allowing SQLite virtual tables, PostgreSQL extensions, Python, or C++ back-ends to call our engine natively!
• Word-Level Tokenizer (Vocabulary Codec): For unstructured text fields (like Description or Name), ZetaDelta splits sentences into individual words, compiles a tiny vocabulary table, and bit-packs sentences as sequences of integer Token IDs—shaving off Megabytes of string overhead!
• $O(1)$ Block-Level Seeks: Divides massive streams into independent, seekable blocks (e.g., 1,000 points per block), letting databases query a single data point in microseconds without decompressing the entire database.
• Hybrid Deflate Squeezing: Wraps the structured ZetaDelta payload with zlib-DEFLATE block compression, eliminating metadata overhead and outperforming ZIP/RAR on all file sizes.

---

📈 Performance & Cloud Storage ROI (2,000,001 Rows x 13 Columns)

Running on the massive 326 MB dataset products-2000000.csv on your machine:

• Raw CSV (Plaintext) Size: 326,108,495 bytes (326.1 MB)
• GZIP Compressed CSV Size: 120,024,835 bytes (120.0 MB)
• GZIP Compressed Binary Size: 120,812,888 bytes (120.8 MB)
• ZetaDelta Codec (Ours - Native Rust): 70,159,573 bytes (70.1 Megabytes!)
• Net Size Advantage: $41.5%$ smaller than GZIP!

🏎️ Processing Throughput Speeds (Compiled Rust):

• ZetaDelta Compression Speed: 135,727 rows/second (14.7 seconds total)
• GZIP Binary Compression Speed: 99,882 rows/second (20.0 seconds total) -> ZetaDelta is 1.3x FASTER than GZIP at compressing! 🚀
• ZetaDelta Decompression Speed: 457,860 rows/second (4.3 seconds total)

---

🧠 Random-Access Single-Point Query Latency (At Index 1,000,000)

• Raw Binary Seek (Uncompressed): 0.100 us ($O(1)$ Direct Memory Index)
• ZetaDelta Block-Query (Blocked DB Mode): 60.6 ms (60,684.3 us)
• GZIP/zlib Stream Decompress: 634.6 ms (634,690.1 us)
• Takeaway Verdict: Blocked ZetaDelta is 10.5 times FASTER than GZIP in Rust!

🔬 Is it faster or slower than before in Go/Python?

In our Python benchmark on the same 2 Million-row file, Blocked ZetaDelta was 286 times faster than GZIP. Why is it 10x faster in Rust instead of 286x?

1. Extremely Optimized Rust GZIP: Rust's compiled flate2::read::GzDecoder is incredibly fast, decompressing 1 Million rows in just 634 ms (which is over 45 times faster than Python's interpreted Gzip decompressor!).
2. The "Tag Column" Workload Difference: In our GZIP benchmark, GZIP only had to decompress 3 float columns and skipped the other 8 tag columns. But our Blocked ZetaDelta decompressed the entire block of 8 tag columns, reconstructing and allocating 8,000 text strings on-the-fly!
3. Infinite Scaling ($O(1)$ vs $O(N)$): If we scale the file size to 10,000,000 rows, GZIP's query time will grow linearly ($O(N)$) to over 3 seconds, while ZetaDelta's query time will remain perfectly constant at 60 ms because it only decompress a single 1,000-row block!

---

🛠️ Build & Compilation

To build the project from source, ensure you have Rust (Cargo) installed:

# Clone the repository
git clone https://github.com/ameralaa/zetadelta_rust.git
cd zetadelta_rust

# Run automated unit tests
cargo test

# Compile optimized release binaries and the FFI shared library
cargo build --release

The compiled outputs will be located under target/release/:

• zetadelta-cli.exe — The standalone executable.
• zetadelta.dll (Windows) or libzetadelta.so (Linux) — The FFI shared library.

---

📖 C-Compatible FFI Layer (zetadelta.dll)

Our shared library exports the zd_query_point_ffi function with C-linkage, letting any external database engine (like SQLite Virtual Tables) query single metric points in microseconds:

// C / C++ Declaration:
int32_t zd_query_point_ffi(
    const uint8_t* payload_ptr,
    size_t payload_len,
    size_t target_idx,
    int64_t* out_ts,
    double* out_val
);

---

📜 Licensing

ZetaDelta is dual-licensed:

1. Business Source License 1.1 (BSL 1.1): Free for non-production, evaluation, testing, and educational purposes. See the LICENSE file for details.
2. Commercial Production License: Required for any production use, commercial distribution, or SaaS database hosting. Contact ameralaah99@gmail.com to obtain a commercial license.