Skip to content

Fix div rounding when the first quotient word has leading zeros - #575

Open
bardrop wants to merge 1 commit into
ruby:masterfrom
bardrop-oss:fix-div-rounding-leading-zero-word
Open

bardrop wants to merge 1 commit into
ruby:masterfrom
bardrop-oss:fix-div-rounding-leading-zero-word

Conversation

@bardrop

@bardrop bardrop commented Nov 18, 2026

Copy link
Copy Markdown

div(value, digits) and quo(value, digits) can drop a nonzero remainder, so the result is rounded as if it were exact:

BigDecimal.mode(BigDecimal::ROUND_MODE, :up)
BigDecimal(6).div(BigDecimal("0.66"), 27)
# => 0.909090909090909090909090909e1 (expected ...910e1)
BigDecimal("-495e-18").div(BigDecimal("3#{'9' * 40}e23"), 9)
# => -0.12375e-78 (expected -0.123750001e-78)

BigDecimal.mode(BigDecimal::ROUND_MODE, :half_even)
BigDecimal("1.000000005").div(BigDecimal("0.#{'9' * 40}"), 9)
# => 0.1e1 (expected 0.100000001e1, the true value is 1.0000000050000...0001...)

The same calls with 26 or 28 digits are right.

Cause

After VpDivd, BigDecimal_div2 marks a nonzero remainder by bumping frac[idx], which should be the first word after the requested digits. It uses idx = roomof(ix, BASE_FIG), which is only true when frac[0] holds 9 digits. For 9.0909... the words are 000000009 090909090 ..., so 27 digits end inside frac[3], not before it. frac[3] is nonzero, nothing is bumped, and the digits after the cut are all zero, so VpLeftRound sees an exact value.

Fix

Count the leading zeros of frac[0] and use roomof(ix + leading_zeros, BASE_FIG). idx is still inside cv (it holds ix + 1 + 2*BASE_FIG - 1 digits, and leading_zeros <= 8).

Testing

  • New cases in test_div_rounding_with_small_remainder (:up, :half_down, :half_even) fail on master and pass with this change.
  • bundle exec rake test: 271 tests, 0 failures.
  • I also ran 100,000 random div/quo calls (all 7 rounding modes, 1 to 60 digits, values from 1e-40 to 1e40) against Python's decimal with the same precision and rounding. Master: 62 wrong results (28 :up, 16 :floor, 10 :ceiling, 6 :half_down, 2 :half_even). This branch: 0. Division by zero was left out (it returns Infinity by design).

BigDecimal#div(value, digits) marks a nonzero remainder by bumping
frac[idx], the word after the requested digits. idx was roomof(digits,
BASE_FIG), which is only right when frac[0] has 9 digits. When it has
fewer, frac[idx] still holds requested digits, so the remainder was lost
and the result was treated as exact:

  BigDecimal.mode(BigDecimal::ROUND_MODE, :up)
  BigDecimal(6).div(BigDecimal('0.66'), 27) # => 0.909090909090909090909090909e1 (should end in 910)

Count the leading zeros of frac[0] when picking idx.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant