Opened 13 years ago

Closed 13 years ago

#8699 closed defect (fixed)

dojo.number.round only supports 13 significant digits but the docs say 14

Reported by: Douglas Hays Owned by: Adam Peller
Priority: high Milestone: 1.3
Component: Core Version: 1.3.0b1
Keywords: Cc:
Blocked By: Blocking:


dojo.number.round is documented to support 14 significant digits and should support at least 14 since JavaScript? supports 15, but there's an error in the function that causes only 13 to be supported.

var delta = Math.pow(10, -14 + wholeFigs);

should be

var delta = Math.pow(10, -15 + wholeFigs);

Thus dojo.number.round(923456789012.37,0,2.5) should return 923456789012.25 but it currently returns 923456789012.50 which is wrong. Similarly dojo.number.round(923456789012.37,2) currently returns 923456789012.38 but it should be 923456789012.37 The suggested fix corrects these problems but doesn't break the other sample calculations. Ticket #8692 is dependent on this change.

Change History (2)

comment:1 Changed 13 years ago by Adam Peller

Milestone: tbd1.3

comment:2 Changed 13 years ago by Adam Peller

Resolution: fixed
Status: newclosed

(In [16814]) Support 14 sig figs. Fixes #8699, Refs #7930 !strict

Note: See TracTickets for help on using tickets.