Opened 12 years ago

Last modified 5 years ago

#10868 assigned defect

scaler/common::eq function broken for large values

Reported by: Douglas Hays Owned by: Eugene Lazutkin
Priority: high Milestone: 1.15
Component: Charting Version: 1.4.2
Keywords: Cc:
Blocked By: Blocking:


This function in scaler/common.js doesn't work for large label values:

var eq = function(/*Number*/ a, /*Number*/ b){
        // summary: compare two FP numbers for equality
        return Math.abs(a - b) <= 1e-6 * (Math.abs(a) + Math.abs(b));   // Boolean

If the label values are [1237940000000, 1237940000001, 1237940000002] then the eq function says 1237940000009 matches all 3. The function should have subtracted the label[0].value before comparing. This is causing charts to show duplicate and incorrect labels.

Change History (4)

comment:1 Changed 12 years ago by Eugene Lazutkin

Milestone: tbdfuture
Status: newassigned

Subtracting an arbitrary value cannot help in all cases, e.g., results still can be too big. Probably we need to use a smaller tolerance value, like 1e-10 (need to confirm it with the JavaScript standard), or even make it configurable.

Being practical your solution seems appropriate --- the best is to massage data like you suggested before feeding it to charting, and use custom labels.

comment:2 Changed 12 years ago by jason_hays22

#10868 was fixed as a preq for #8996.

comment:3 Changed 6 years ago by dylan

Milestone: future1.12

comment:4 Changed 5 years ago by dylan

Milestone: 1.131.15

Ticket planning... move current 1.13 tickets out to 1.15 to make it easier to move tickets into the 1.13 milestone.

Note: See TracTickets for help on using tickets.