Skip to content

Skip unparseable gold solutions in cosine_scaled_reward - #734

Open
shaurya416 wants to merge 1 commit into
huggingface:mainfrom
shaurya416:fix-cosine-reward-unparseable-gold
Open

shaurya416 wants to merge 1 commit into
huggingface:mainfrom
shaurya416:fix-cosine-reward-unparseable-gold

Conversation

@shaurya416

Copy link
Copy Markdown

Fixes #732

Problem

get_cosine_scaled_reward returns 1.0 for a completion whenever the gold solution cannot be parsed (rewards.append(1.0) # Skip unparseable examples). 1.0 is the default max_value_correct, so an example with a malformed or missing gold answer gives the model the maximum reward for any output, including one that is plainly wrong. The comment says the example is skipped, but a numeric reward is not a skip.

accuracy_reward handles the same case by returning None, so the trainer skips the example (#566). This change does the same in cosine_scaled_reward.

Change

  • rewards.append(None) instead of 1.0 when len(gold_parsed) == 0, and the return annotation list[Optional[float]] as on accuracy_reward.
  • Two tests in tests/test_rewards.py: an unparseable gold gives [None, None] for completions of different lengths, and in a mixed batch only the unparseable example is None.

Not changed: len_reward has the same shape (correctness.append(True)), but test_unparseable_solution pins its current 0.5, so I left it for a separate decision. The optional follow-up from the issue (dropping extraction_config=[LatexExtractionConfig()] on the gold parse so bare answers stay scorable) is also left out.

Validation

I ran the real tests/test_rewards.py against the real rewards.py with math-verify==0.5.2 and latex2sympy2_extended, from a stub package that replaces only the modules the reward functions under test never call (code_providers, competitive_programming, configs). TestGetRewardFuncs was deselected because it needs the real configs.

new tests (2) existing tests
main 2 failed 45 passed, 1 failed
this change 2 passed 45 passed, 1 failed

The one failing existing test, TestRepetitionPenaltyReward::test_full_repetition_with_language, fails the same way before and after because transformers is not installed in this environment. ruff format --check --line-length 119 passes on the two files.

When the gold solution could not be parsed, cosine_scaled_reward returned 1.0,
the maximum reward, for every completion, so a malformed or missing gold answer
rewarded the model for any output. accuracy_reward already returns None for that
case since huggingface#566, which lets the trainer skip the example. Return None here too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cosine_scaled_reward returns 1.0 for every completion when the gold answer cannot be parsed (should be None, as accuracy_reward does since #566)

1 participant