Skip to content

HDDS-17820. Add SCM metrics for storage-tier placement failures by requested storage type and reason. - #12079

Draft
devmadhuu wants to merge 1 commit into
apache:HDDS-11233from
devmadhuu:HDDS-17820
Draft

devmadhuu wants to merge 1 commit into
apache:HDDS-11233from
devmadhuu:HDDS-17820

Conversation

@devmadhuu

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

SCM currently logs tier-placement failures but does not expose counters showing why allocation on SSD, DISK or ARCHIVE failed. A write can succeed through fallback, leaving administrators without metrics explaining why the preferred tier could not be used.

This PR adds the following counter to SCMPipelineMetrics for RATIS/standalone allocation:

scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_space"}

The counter uses bounded labels:

Label Values
requestedtier SSD, DISK, ARCHIVE, UNKNOWN
reason no_space, no_suitable_node, other

Reasons are classified using existing SCM result codes:

- no_space: insufficient eligible storage of the requested type. This includes space/type eligibility failures and does not necessarily mean the physical disks are full. - no_suitable_node: insufficient healthy/suitable nodes or placement restrictions such as pipeline limits. - other: unclassified failures, including ordinary I/O errors and readiness timeouts.

A failure is counted once after allocation on that tier exhausts the existing retries. Successful recovery on the same tier does not increment the counter.

For example, suppose ten HOT block allocations prefer SSD. Six succeed on SSD, three fail its space/type check, and one fails suitable-node selection. If all four unsuccessful SSD allocations succeed through DISK fallback, administrators see:

scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_space"} 3
scm_pipeline_metrics_tier_placement_failures{requestedtier="SSD",reason="no_suitable_node"} 1
scm_pipeline_metrics_tier_placement_failures{requestedtier="DISK",reason="no_space"} 0

These counters measure failed tier allocations, rather than failed keys or bytes. They reset with SCM metrics registration. All tier/reason combinations are exported, including zeros.

Existing aggregate metrics, allocation decisions, retry behavior, exception messages and fallback rules remain unchanged. EC related tier allocation metrics will be done in follow up PR.

What is the link to the Apache JIRA?

HDDS-17820 (https://issues.apache.org/jira/browse/HDDS-17820)

How was this patch tested?

  • Added TestSCMTierPlacementMetrics for reason classification, tier separation, UNKNOWN handling, Prometheus output, concurrent updates and registration/reset behavior.
  • Extended TestWritableRatisContainerProvider to verify terminal failure counting, successful recovery, readiness failures and unchanged exceptions.
  • Extended TestBlockManager to verify permitted/disabled fallback and separate accounting when both preferred and fallback tiers fail.
  • SCM allocation, pipeline and placement regression run: 508 passed, 19 existing skips, no failures or errors.
  • Existing pipeline-metrics mini-cluster tests: 3 passed. An existing pipeline-command test also passed.

@devmadhuu

Copy link
Copy Markdown
Contributor Author

@anuragp010 Another PR for SCM tiering allocation metrics. Kindly have a look.

@github-actions github-actions Bot added the scm label Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant