Repository navigation
Conversation
Edges poll /v1/n9e/statistic?name=target every 9s and, whenever it changes, pull /v1/n9e/targets. Both hit the DB on the center: count(*)/max(update_at) over the target table (no index on update_at) and a full load of target plus target_busi_group. With 100k+ targets and a dozen edges reloading at the same moment, these show up as multi-second target queries. Serve both from the center's TargetCache instead. The cache holds the same rows TargetGetsAll returns and is refreshed on the same statistic, so edges see at most one extra cache period (~9s) of lag. - TargetCacheType: write stat together with targets under the lock; add Stat() and Snapshot() (shallow copies taken under the read lock, so they can be encoded while BeatTime/meta refreshes mutate the cached objects). - targetGetsByService / statistic(name=target): answer from the cache, falling back to the DB when no cache is wired.
…roup reset Edge: the stat and the target list are two requests that poster may send to different centers, each now answering from its own TargetCache. Storing a stale list under a newer center's stat latched it until the next target change. Edges/alerts now store the stat derived from the list they received (len, max(update_at)), so a stale list is reloaded on the next poll. TargetOverrideBgids: resetting a target to no busi group deleted the bindings but returned before bumping target.update_at, so TargetCache (and the collect static gate in plus that now relies on its stat) kept the old GroupIds. Bump update_at when bindings were actually removed; skip it otherwise, since heartbeats with overwrite_gids and an empty gid call this on every beat.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every edge/alert instance polls
/v1/n9e/statistic?name=targetevery 9s and, when the statistic changes, pulls/v1/n9e/targets. On the center both go straight to the DB:SELECT count(*), max(update_at) FROM target: a full scan each time, sinceupdate_athas no index.TargetGetsAll: loads the wholetargetandtarget_busi_grouptables and serializes all of it.In a deployment with 100k+ targets and ~14 edges, every target change makes all edges reload at once. Each reload took 2–10s, and target-table queries hit a P99 of 2–3s.
Change
Both endpoints now answer from the center's
TargetCache. It holds the same rowsTargetGetsAllreturns and refreshes on the same statistic.memsto.TargetCacheTypeStat()Snapshot(): shallow copies taken under the read lock, so the response can be encoded whilerefreshBeatTime/refreshHostMetaskeep mutating the cached objectstargetGetsByServiceandstatistic(name=target) read from the cache, and fall back to the DB when no cache is wired.The trade-off is that edges may see target changes up to one more center cache period (~9s) later.
Multi-center consistency
The stat and the list are two requests, and
poster.GetByUrlspicks a random center for each. With several centers, an edge could get the new stat from a center that has reloaded and the old list from one that has not, then store that list under the new stat. Once both centers caught up, the stat no longer changed and the stale list stayed until the next target change.!IsCenter) now store the stat derived from the list they received (len, maxupdate_at). A stale list then differs from the centers' stat and is reloaded on the next poll, at the cost of at most one extra reload.Resetting a target to no busi group
TargetOverrideBgidswith emptybgidsdeleted the bindings but returned before bumpingtarget.update_at, soTargetCachekept the oldGroupIds. This affects edges and the collect static gate in n9e-plus#1417, which now relies onTargetCache.Stat().update_atwhen bindings were actually removed. It does not bump otherwise, because heartbeats withoverwrite_gids=trueand an emptygidcall it on every beat, and an unconditional bump would force a full reload each cycle.flashcatcloud/n9e-pluswill use the newTargetCache.Stat()/Snapshot()to stop re-running per-collect-rule target SQL; that PR depends on this one.Tests
center/router/router_target_service_test.gomemsto/target_cache_test.go: an edge given a newer stat and an older list stores the list's stat and reloads on the next pollmodels/target_busi_group_test.go: resetting to no group bumpsupdate_atonce, and a repeated reset leaves it unchangedgo test -race ./memsto/ ./center/router/andgo vetpass.