Skip to content

[dbnode] - Add filesystem available space reporting#3450

Open
shrek wants to merge 5 commits into
m3db:masterfrom
shrek:add-disk-metrics
Open

[dbnode] - Add filesystem available space reporting#3450
shrek wants to merge 5 commits into
m3db:masterfrom
shrek:add-disk-metrics

Conversation

@shrek

@shrek shrek commented Apr 25, 2021

Copy link
Copy Markdown

What this PR does / why we need it:
Fixes #3400
Please refer to the discussion in the issue. In summary, this PR adds metrics
for available space in the filesystem, and the sizes of files in different m3db
directories. It can be used to alert on filesystem available space thresholds.
The added metrics look like this:

# TYPE filesystem_avail_bytes gauge
filesystem_avail_bytes 4.2259341312e+10
# HELP filesystem_commitlogs_bytes filesystem_commitlogs_bytes gauge
# TYPE filesystem_commitlogs_bytes gauge
filesystem_commitlogs_bytes 40
# HELP filesystem_data_bytes filesystem_data_bytes gauge
# TYPE filesystem_data_bytes gauge
filesystem_data_bytes 66304
# HELP filesystem_index_bytes filesystem_index_bytes gauge
# TYPE filesystem_index_bytes gauge
filesystem_index_bytes 4310
# HELP filesystem_num_dir_api_errors filesystem_num_dir_api_errors counter
# TYPE filesystem_num_dir_api_errors counter
filesystem_num_dir_api_errors 1.829242e+06
# HELP filesystem_num_fs_api_errors filesystem_num_fs_api_errors counter
# TYPE filesystem_num_fs_api_errors counter
filesystem_num_fs_api_errors 0
# HELP filesystem_snapshots_bytes filesystem_snapshots_bytes gauge
# TYPE filesystem_snapshots_bytes gauge
filesystem_snapshots_bytes 1.8173384e+07
# HELP filesystem_total_bytes filesystem_total_bytes gauge
# TYPE filesystem_total_bytes gauge
filesystem_total_bytes 6.2725623808e+10

Special notes for your reviewer:

Does this PR introduce a user-facing and/or backwards incompatible change?:

NONE

Does this PR require updating code package or user-facing documentation?:

NONE

shrek added 4 commits April 4, 2021 13:38
Currently, there are no metrics reporting on the
amount of space available in the filesystem directory
path used by the dbnode. Add a reporter which
reports on the available space. This metric could be
used for alerting purposes if the available space
crosses thresholds.

Example metrics reported:

filesystem_avail_bytes 4.3406266368e+10
filesystem_num_api_errors 0
filesystem_total_bytes 6.2725623808e+10

Which matches with the output of df
1K-blocks      Used Available
61255492  15725236  42388932
@CLAassistant

CLAassistant commented Apr 25, 2021

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codecov

codecov Bot commented Sep 11, 2023

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.85507% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.4%. Comparing base (1843061) to head (837f895).
⚠️ Report is 665 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (1843061) and HEAD (837f895). Click for more details.

HEAD has 84 uploads less than BASE
Flag BASE (1843061) HEAD (837f895)
collector 20 9
aggregator 22 9
m3em 20 9
cluster 20 9
msg 6 4
dbnode 17 9
query 15 5
x 11 5
m3ninx 16 5
metrics 5 4
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #3450     +/-   ##
=========================================
- Coverage    72.4%    64.4%   -8.0%     
=========================================
  Files        1100     1100             
  Lines      102443   101703    -740     
=========================================
- Hits        74203    65579   -8624     
- Misses      23143    31312   +8169     
+ Partials     5097     4812    -285     
Flag Coverage Δ
aggregator 69.6% <ø> (-7.3%) ⬇️
cluster 65.7% <ø> (-19.2%) ⬇️
collector 84.3% <ø> (ø)
dbnode 75.6% <71.4%> (-3.4%) ⬇️
m3em 68.6% <ø> (-5.8%) ⬇️
m3ninx 62.8% <ø> (-10.8%) ⬇️
metrics 19.8% <ø> (ø)
msg 74.4% <ø> (+<0.1%) ⬆️
query 52.5% <ø> (-14.6%) ⬇️
x 63.3% <91.9%> (-17.3%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1843061...837f895. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[dbNode] Add filesystem available space reporting

2 participants