[NGSOK-1826] Metrics for Spark History Server#47
Merged
Conversation
giggsoff
force-pushed
the
feature/3/NGSOK-1826
branch
from
044d298 to
172ab38
Compare
Start an optional MetricsSystem for the History Server, gated by the new spark.history.metrics.enabled config (disabled by default so existing deployments are unaffected unless opted in). When enabled it exposes the existing application cache counters and a small set of cheap server-wide gauges (HistoryServerSource) through the configured sinks, e.g. the Prometheus servlet at /metrics/applicationHistory/prometheus. Also register the ApplicationCache counters/timer into its MetricRegistry under plain-noun names so the metrics system's own count suffix is not duplicated; toString output is unchanged. JVM heap/GC metrics for the History Server process are available opt-in via the standard source config (applicationHistory.source.jvm.class=JvmSource), exactly like the other Spark instances; covered by the metrics test. Signed-off-by: Petr Fedchenkov <giggsoff@gmail.com>
Expose byte-valued disk and in-memory store usage of the FsHistoryProvider through the History Server metrics system. A new getMetricsSources hook on ApplicationHistoryProvider (default empty) lets a provider contribute its own metric sources without leaking internals; FsHistoryProvider supplies FsHistoryProviderSource with diskStore/memoryStore byte gauges, each guarded so it only appears when the backing store is active. Signed-off-by: Petr Fedchenkov <giggsoff@gmail.com>
giggsoff
force-pushed
the
feature/3/NGSOK-1826
branch
from
172ab38 to
ee77a16
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wire a metrics system into the Spark History Server, which previously had none, exposing its internal state to the standard Spark metrics sinks (e.g. the Prometheus servlet).