feat(core): add optional no-op KV cache connector seam

[marcusacosta]

Summary

• Adds plumbing for an optional extra KV-cache storage layer on top of today's local prefix cache (Proposal: additive KV cache connector seam at paged-attention block pool #2308).
• Introduces a KvCacheConnector hook (default NoopKvCacheConnector does nothing) so later backends (memory/disk/S3/etc.) can plug in without changing normal users.
• Calls that hook when the engine looks up, stores, or evicts prefix-cache blocks (get_computed_blocks, cache_blocks, and block-pool eviction).
• Keeps the default path local-only: same constructors as before (with_connector is opt-in); no behavior change unless a real connector is installed.
• This PR is only the connection points. It does not add disk, cloud, or network KV storage yet.

Notes for reviewers

• If lookup_blocks returns block IDs, those IDs must already exist in the local block pool (a future connector would download/hydrate first, then return local IDs).
• Happy to adjust the trait shape to maintainer preference, as noted in Proposal: additive KV cache connector seam at paged-attention block pool #2308.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 24         4497         3151          790          556
 CSS                       3          282          252            6           24
 CUDA                    124        26447        21760         1707         2980
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          578          563           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        17989        17986            0            3
 Makefile                  1           18           16            0            2
 MDX                      36         6285            0         4616         1669
 Metal Shading Lan|       37        14416        11408         1136         1872
 PowerShell                1          657          571           31           55
 Python                  151        12539        10418          480         1641
 Shell                     3         1071          852          115          104
 Plain Text               53        10687            0         9209         1478
 TOML                     28         1389         1209           39          141
 TypeScript               11         1649         1410           66          173
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                273        12055            0         8981         3074
 |- BASH                  24          300          221           47           32
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6120          306          923
 |- Rust                  62         3840         2849          394          597
 |- TOML                   7           92           78            0           14
 (Total)                            23940         9570         9728         4642
─────────────────────────────────────────────────────────────────────────────────
 Rust                    705       354276       318622         5547        30107
 |- Markdown             416         9915          452         8275         1188
 (Total)                           364191       319074        13822        31295
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1969         1826           51           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2894         2597           76          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1515       490614       401788        41895        46931
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[EricLBuehler]

Hey @marcusacosta! Thanks for the PR. Perhaps we can add an example of an external KV cache manager if this is the purpose?

[marcusacosta]

@EricLBuehler

Yes, the seam is meant for external KV tiers. I’ll add a small example external connector (in-process / reference manager) that shows store/lookup/evict through KvCacheConnector, without a full cloud backend. Sound good?

[marcusacosta]

Added a reference external manager:

• InMemoryKvCacheConnector in mistralrs-core (process-local hash -> block-id index outside the block pool map)
• runnable example: cargo run -p mistralrs --example kv_cache_connector

This shows store/lookup/hit through the seam. It does not offload tensor bytes to disk/S3 yet (that still needs a hydrate path). Happy to adjust the example shape if you want something different.

[EricLBuehler]

Hmm, I was thinking something more along the lines of using the manager as one would in a realistic setting, in contrast to demonstrating the manager in isolation. Is this what you had in mind as well?

[marcusacosta]

Updated to a realistic path:

• PagedAttentionMetaBuilder::with_kv_cache_connector(...) installs the connector into the live paged-attention scheduler
• example now uses ModelBuilder + paged attention and runs two chat turns with a shared prefix, printing connector store/lookup/hit stats

cargo run --release -p mistralrs --example kv_cache_connector