Skip to content

Improve performance with multiple plz-outs #3629

Closed

Description

@peterebden

We sort of have an implicit soft assumption that there's generally one plz-out per repo. It works with multiple, especially recently where the cache has gained locking,

This is quite a bit less true these days now people are more heavily using git worktrees, often because of / assisted by agents. In these cases there are multiple of them which apparently take a lot of on-disk space. This is not actually trivial to calculate though - it appears bigger than it is because they share items from the cache which are hardlinked.
However, this doesn't work super well, because the cache tends to get swamped at this point so we have to churn cleaning it a lot. Cleaning also isn't actually super effective because often we are unlinking the cache but not the linked version, which saves nothing. The high / low watermark defaults also tend to be too small in a lot of cases.

I think this can be improved with some work which doesn't require massive re-working of everything (e.g. making everything content-addressed). Rough plan:

  1. Count cache size only by items that are not linked elsewhere and evict older items.
  2. Take the disk size / free space into account when cleaning.
  3. Synchronise across repos so we only run one cleaner at a time.
  4. Maybe some kind of info / visualisation of the current state of it?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions