Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#10898 closed enhancement (fixed)

[ccla][patch] Nulls in data series should make gaps, instead of being treated as zeroes

Reported by: Adam Peller Owned by: Eugene Lazutkin
Priority: high Milestone: 1.5
Component: Charting Version: 1.4.0
Keywords: Cc: deanw, davec
Blocked By: Blocking:

Description (last modified by Adam Peller)

Currently, in a line chart, when a null appears in the data series the line drops to zero at that point and then resumes. A null generally means that data is "missing", rather than zero. The interpretation of nulls as zero was probably introduced with bars and columns, and the effect on lines not fully considered. It would be better if a null in the data series caused a "gap" in the line, rather than being treated as zero. This feature requires splitting the data series into "dense" (null-free) ranges, and rendering each as a distinct gfx shape, taking care that the same line plot attributes are applied to multiple ranges from the same series (eg sequential colour allocation). Also, the automatic ("natural") adjustment of axes should ignore nulls.

Test: a test is included which illustrates both this item and #10897 (test_lines.html)

Attachments (3)

100324-nulls.patch (20.2 KB) - added by Adam Peller 10 years ago.
patch from Dave Clark (IBM, CCLA)
alternate-10898.patch (3.8 KB) - added by Scott Andrews 10 years ago.
patch from Scott Andrews, VMware CCLA portions Copyright 2010 VMware Dojo is available under *either* the terms of the modified BSD license *or* the Academic Free License version 2.1. As a recipient of Dojo, you may choose which license to receive this code under (except as noted in per-module LICENSE files). Some modules may not be the copyright of the Dojo Foundation. These modules contain explicit declarations of copyright in both the LICENSE files in the directories in which they reside and in the code itself. No external contributions are allowed under licenses which are fundamentally incompatible with the AFL or BSD licenses that Dojo is distributed under.
fresh-nulls.patch (54.7 KB) - added by davec 10 years ago.
fresh patch from Dave Clark (IBM, CCLA)

Download all attachments as: .zip

Change History (21)

Changed 10 years ago by Adam Peller

Attachment: 100324-nulls.patch added

patch from Dave Clark (IBM, CCLA)

comment:1 Changed 10 years ago by Adam Peller

Description: modified (diff)

comment:2 Changed 10 years ago by Adam Peller

patch includes changes from #10897

comment:3 Changed 10 years ago by Eugene Lazutkin

Summary: Nulls in data series should make gaps, instead of being treated as zeroes[ccla][patch] Nulls in data series should make gaps, instead of being treated as zeroes

comment:4 Changed 10 years ago by Eugene Lazutkin

Status: newassigned

comment:5 Changed 10 years ago by Eugene Lazutkin

This patch clashes with #10856. Need to reconcile. :-(

comment:6 Changed 10 years ago by Scott Andrews

We ran into the same issue but took a different approach to resolution. Instead of drawing separate shapes for each dense range, we use a single poly line that skips data point that are undefined. To make this simple, we separate the datapoint array into separate arrays for the line and markers. The markers array contains null values as placeholders, while the line array only hold valid data points.

This patch (alternate-10898.patch) is less invasive then what has already been submitted and only conflicts with #10856 on one line that should be trivial to resolve.

Changed 10 years ago by Scott Andrews

Attachment: alternate-10898.patch added

patch from Scott Andrews, VMware CCLA portions Copyright 2010 VMware Dojo is available under *either* the terms of the modified BSD license *or* the Academic Free License version 2.1. As a recipient of Dojo, you may choose which license to receive this code under (except as noted in per-module LICENSE files). Some modules may not be the copyright of the Dojo Foundation. These modules contain explicit declarations of copyright in both the LICENSE files in the directories in which they reside and in the code itself. No external contributions are allowed under licenses which are fundamentally incompatible with the AFL or BSD licenses that Dojo is distributed under.

comment:7 Changed 10 years ago by Adam Peller

Cc: deanw davec added

comment:8 Changed 10 years ago by davec

Scott, am I right in thinking your patch will cause the line to extend across nulls? ie, if a series was [ 1, 2, null, 4, 5 ] then your line would have markers (if enabled) at (1,1), (2,2), (4,4) and (5,5), and line segments connecting (1,1) to (2,2), (2,2) to (4,4) and (4,4) to (5,5)? This differs slightly from my patch, which would omit the line segment from (2,2) to (4,4) -- ie, the null would cause a line break. However, we have thought of scenarios where having the line extend across a null might make more sense --- should this be an option controlled somewhere? If so, how?

comment:9 Changed 10 years ago by Scott Andrews

@davec

You are correct that the line spans across null values. In our use case, it is common to have a single data point surrounded by nulls. To avoid a free standing data point, we span the null values.

My original idea was to use a dashed line for the segment spanning null values. However, this required separate shapes for each segment, which lead to chaos with tensioning. For simplicity, we went with a single shape.

I'm not opposed to an option to allow null spanning or not. Although the render method is already fairly bloated. The optionalParams bag is the best place to define the value.

comment:10 Changed 10 years ago by Eugene Lazutkin

Dave is right, that the render method of Default is overloaded and complex already to support different strategies.

I am tempted to accept Dave's patch for the Default plot, and spin off Scott's patch as an alternative implementation of this plot but under a different name.

The truth is that both approaches are perfectly valid in different situations. For example, while Dave's approach is very much conventional for line/area plots, it doesn't work well for stacked lines/areas, while Scott's approach deals with missing points and can be easily adapted for that.

comment:11 Changed 10 years ago by davec

ok, I will post a fresh patch once #10856 is in (so as to avoid clashes). Also, Eugene makes a good point about stacked charts: in stacked charts of all types, a null should be treated effectively as a zero but with no actual rendering of even a zero-height segment in the stack (bars/columns). In ordinary bar/column/pie charts, a null should simply suppress rendering of the bar/column/pie segment. This all used to work, but some recent changes to allow data-point-custom style overrides have inadvertently broken this (nulls in series for bars/columns/etc now cause abrupt termination and incomplete rendering) -- that's easy to fix, and I'll include that in the patch too.

It would be good to include Scott's approach (bridging across nulls) at some point, and I do really like the idea of a dashed line -- the use case we found was a chart with lines and no markers, and we were concerned that "bridging" across a null was indistinguishable from a true data point at the intermediate value. A dashed line would resolve that, if we could work on the right way to render it. We'd need a way to have tension on a line comprising multiple line segments rendered with different stroke styles.

comment:12 Changed 10 years ago by Scott Andrews

+1

comment:13 Changed 10 years ago by Eugene Lazutkin

#10856 was committed in [21862]. You can proceed with the patch.

Changed 10 years ago by davec

Attachment: fresh-nulls.patch added

fresh patch from Dave Clark (IBM, CCLA)

comment:14 Changed 10 years ago by davec

ok, I have posted a fresh patch to enable nulls to be handled in various chart types. The patch comprises the logic as before for lines and areas to include a "gap" (Default.js), and as before amends the axis range logic (common.js) so that nulls do not extend the axis range to include zero. This patch also includes small adjustments to various other chart types to ensure nulls are handled without error.

The rationale I have applied is that where there are individual rendered elements (bars, columns, pie segments, candlesticks, bubbles) the rendering of the element is simply omitted when the series contains a null; for the stacked types (stacked bars, columns, lines, areas) the null continues to be treated as an effective zero for accumulating the values; for the stacked bars and columns no actual bar/column element is rendered when the series contains a null; for stacked lines and areas the line/area continues as if the null were a zero. We could easily apply the same "gap" approach for stacked lines/areas, but I was not sure this was the best approach --- by including a line/area into a stack I think we're basically forcing it to take a numerical value, ie zero (as we do with NaNs? in this case too).

I've included a simple test HTML to check the behaviour on nulls of the various chart types.

comment:15 Changed 10 years ago by Eugene Lazutkin

(In [21927]) charting: special treatment for nulls across all plot types, thx Dave Clark!, !strict, refs #10898.

comment:16 in reply to:  14 ; Changed 10 years ago by Eugene Lazutkin

Resolution: fixed
Status: assignedclosed

Replying to davec:

While applying the patch I had some conflicts, so I had to resolve them manually. Please review [21927] to make sure that I didn't screw up anything.

comment:17 in reply to:  12 Changed 10 years ago by Eugene Lazutkin

Replying to scothis:

+1

Could you open another ticket for an alternative treatment of nulls with your patch attached?

comment:18 in reply to:  16 Changed 10 years ago by davec

Please review [21927] to make sure that I didn't screw up anything.

Thanks Eugene, looks perfect to me.

Note: See TracTickets for help on using tickets.