Skip to content

perf(center): serve edge target sync from the center TargetCache - #3456

Open
710leo wants to merge 2 commits into
mainfrom
perf/target-serve-from-cache
Open

710leo wants to merge 2 commits into
mainfrom
perf/target-serve-from-cache

Conversation

@710leo

@710leo 710leo commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Problem

Every edge/alert instance polls /v1/n9e/statistic?name=target every 9s and, when the statistic changes, pulls /v1/n9e/targets. On the center both go straight to the DB:

  • SELECT count(*), max(update_at) FROM target: a full scan each time, since update_at has no index.
  • TargetGetsAll: loads the whole target and target_busi_group tables and serializes all of it.

In a deployment with 100k+ targets and ~14 edges, every target change makes all edges reload at once. Each reload took 2–10s, and target-table queries hit a P99 of 2–3s.

Change

Both endpoints now answer from the center's TargetCache. It holds the same rows TargetGetsAll returns and refreshes on the same statistic.

  • memsto.TargetCacheType
    • writes the stat under the lock together with targets
    • new Stat()
    • new Snapshot(): shallow copies taken under the read lock, so the response can be encoded while refreshBeatTime/refreshHostMetas keep mutating the cached objects
  • targetGetsByService and statistic (name=target) read from the cache, and fall back to the DB when no cache is wired.

The trade-off is that edges may see target changes up to one more center cache period (~9s) later.

Multi-center consistency

The stat and the list are two requests, and poster.GetByUrls picks a random center for each. With several centers, an edge could get the new stat from a center that has reloaded and the old list from one that has not, then store that list under the new stat. Once both centers caught up, the stat no longer changed and the stale list stayed until the next target change.

  • Edges/alerts (!IsCenter) now store the stat derived from the list they received (len, max update_at). A stale list then differs from the centers' stat and is reloaded on the next poll, at the cost of at most one extra reload.

Resetting a target to no busi group

TargetOverrideBgids with empty bgids deleted the bindings but returned before bumping target.update_at, so TargetCache kept the old GroupIds. This affects edges and the collect static gate in n9e-plus#1417, which now relies on TargetCache.Stat().

  • It now bumps update_at when bindings were actually removed. It does not bump otherwise, because heartbeats with overwrite_gids=true and an empty gid call it on every beat, and an unconditional bump would force a full reload each cycle.

flashcatcloud/n9e-plus will use the new TargetCache.Stat()/Snapshot() to stop re-running per-collect-rule target SQL; that PR depends on this one.

Tests

  • center/router/router_target_service_test.go
    • both endpoints are served from the cache with no DB attached
    • the stat matches the loaded targets
    • snapshot copies are isolated from the cache
  • memsto/target_cache_test.go: an edge given a newer stat and an older list stores the list's stat and reloads on the next poll
  • models/target_busi_group_test.go: resetting to no group bumps update_at once, and a repeated reset leaves it unchanged
  • go test -race ./memsto/ ./center/router/ and go vet pass.

Edges poll /v1/n9e/statistic?name=target every 9s and, whenever it changes,
pull /v1/n9e/targets. Both hit the DB on the center: count(*)/max(update_at)
over the target table (no index on update_at) and a full load of target plus
target_busi_group. With 100k+ targets and a dozen edges reloading at the same
moment, these show up as multi-second target queries.

Serve both from the center's TargetCache instead. The cache holds the same rows
TargetGetsAll returns and is refreshed on the same statistic, so edges see at
most one extra cache period (~9s) of lag.

- TargetCacheType: write stat together with targets under the lock; add Stat()
  and Snapshot() (shallow copies taken under the read lock, so they can be
  encoded while BeatTime/meta refreshes mutate the cached objects).
- targetGetsByService / statistic(name=target): answer from the cache, falling
  back to the DB when no cache is wired.
…roup reset

Edge: the stat and the target list are two requests that poster may send to
different centers, each now answering from its own TargetCache. Storing a
stale list under a newer center's stat latched it until the next target change.
Edges/alerts now store the stat derived from the list they received
(len, max(update_at)), so a stale list is reloaded on the next poll.

TargetOverrideBgids: resetting a target to no busi group deleted the bindings
but returned before bumping target.update_at, so TargetCache (and the collect
static gate in plus that now relies on its stat) kept the old GroupIds. Bump
update_at when bindings were actually removed; skip it otherwise, since
heartbeats with overwrite_gids and an empty gid call this on every beat.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant