Opened 9 years ago

Closed 6 years ago

#11370 closed defect (patchwelcome)

DataGrid leaks memory when scrolling and does not respect keepRows attribute.

Reported by: wgrant Owned by: Bryan Forbes
Priority: blocker Milestone: tbd
Component: DojoX Grid Version: 1.4.3
Keywords: Cc: Evan, Jared Jurkiewicz, ptwobrusell
Blocked By: Blocking:

Description

I have connected my DataGrid? to a QueryReadStore?. As the grid is scrolled through a large set of results, the memory used by the browser (vendor and version irrelevant) grows continuously. I have also set the keepRows property to keep the DataGrid?'s cache down, but I can see by the DataGrid?._by_idx object that the keepRows property is not being honored.

I am using the DataGrid? in a query application that allows users to set the filter fields which then sets the query on the DataGrid? and loads new data. The common use case has the window open for most of the day while users just change their queries. This also raises their browser memory as even though I have only 20 rows loading into the table at a time, the rows for each consecutive query do not get freed up. User's will start with IE using 30M and end the day with their browsers crashing with their browser using 300M. I have tried everything I can think of including calling _clearData() and destroying the grid and store. I cannot get IE's memory back down without closing and re-opening the browser. I can't suggest this to my 5000+ registered users.

If I cannot get this resolved, I will have to shop for a new toolkit and refactor my whole project. I also use Zend Framework, so I really don't want to do this. I hope there is a solution!

Change History (10)

comment:1 Changed 9 years ago by Adam Peller

a reproducible test case might help

comment:2 in reply to:  1 Changed 9 years ago by wgrant

Replying to peller:

a reproducible test case might help

Of course... Sorry.

The simplest example I can make uses Matthew Russell's ONLamp article

Dojo Goodness, Part 6 (A Million Records in the Grid) http://www.oreillynet.com/onlamp/blog/2008/04/dojo_goodness_part_6_a_million.html ... simply updated to use the new DataGrid?:

<html>
  <head>
    <title>Memory Leak Demo</title>
    <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.4/dijit/themes/nihilo/nihilo.css">
    <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.4/dojo/resources/dojo.css">
    <link rel="stylesheet" href="http://o.aolcdn.com/dojo/1.4/dojox/grid/resources/Grid.css">

    <script
      type="text/javascript"
      src="http://o.aolcdn.com/dojo/1.4/dojo/dojo.xd.js"
      djConfig="parseOnLoad:true"
    ></script>

    <script type="text/javascript">
      dojo.require("dojox.data.QueryReadStore");
      dojo.require("dojox.grid.DataGrid");
      dojo.require("dojo.parser");

      // a simple grid layout that consists of 5 columns
      layout = [
          {name:'ID',field:'id',width:'5'},
          {name:'Item',field:'item',width:'auto'},
          {name:'Quantity',field:'quantity', width:5},
          {name:'Cost',field:'cost', width: 5 },
          {name:'Total',field:'total', width: 5}
      ];

      dojo.addOnLoad(function(){
        dojo.byId('grid').keepRows = 50;  // set to 75 be default anyway.
      });
    </script>
    </head>
    <body class="nihilo" id="body">
        <!-- The dojo.data API implementation that talks to the server -->
        <div dojoType="dojox.data.QueryReadStore" id="store" jsId="store" url="/data"></div>

        <!-- The grid, which relies on its DojoData abstraction for info -->
        <div style="height:300px; width:400px;" 
             id="grid"
             jsId="grid"
             dojoType="dojox.grid.DataGrid"  
             structure="layout" 
             delayScroll="true" 
             rowsPerPage="20" 
             store="store"
        ></div>

    </body>
</html>

He uses the following simple web server written in Python to deliver dubious amounts of random data for the grid to display:

"""
An ultra-simple web server that provides slices of a very large (mock) data
source for a dojox.grid.Grid client that uses a dojox.data.QueryReadStore
to page the data on demand
"""

import cherrypy #do an "easy_install cherrypy" to get it
from cherrypy.lib.static import serve_file

import demjson #do an "easy_install demjson" to get it
import os
from random import randint #for building up mock data

json = demjson.JSON(compactly=False)
jsonify = json.encode

NUM_ITEMS = 1000000

class Content:
    def __init__(self):
        """
        maybe you would call out to a db with some sql to get some data
        based on the query string that comes into /data. for now, we'll
        build up some static data to use.
        """

        self.items = []

        possible_item_names = ["Foo", "Bar", "Baz", "Bop"]
        id=0
        for i in xrange(NUM_ITEMS):
            self.items.append({
                    'id' : id,
                    'item' : possible_item_names[randint(0,3)],
                    'quantity' : randint(0,10),
                    'cost' : randint(0,100)
            })
            id +=1

        #keep track of sort order b/c sorting is expensive...
        self.current_sort_order = ""

    @cherrypy.expose
    def data(self, **kw):
        """
        serve up the data via http://localhost:8000/data

        kw will contain whatever is in your store's query. by default
        the query string will come across as something like:
        ?name=*&start=0&count=20 to populate the table 

        note: you may get into trouble if you have multiple users
        trying to access this url and changing the sort order of items
        all at the same time (but relax, this is just a little demo.)
        """

        #sorting the items by values for a given dictionary key...
        if kw.get('sort') and self.current_sort_order != kw.get('sort'):
            if kw['sort'][0] == '-': #descending order, slice off the -
                self.items.sort(lambda m,n:cmp(m.get(kw['sort'][1:]), n.get(kw['sort']    [1:])),reverse=True)
           else: #ascending order
                self.items.sort(lambda m,n:cmp(m.get(kw['sort']), n.get(kw['sort'])))
            self.current_sort_order = kw['sort']

        #slicing the data...
        start = int(kw['start'])
        end = start + int(kw['count'])

        #serving up the slice of interest as well as the total size
        return jsonify({'numRows':NUM_ITEMS, 'items':self.items[start:end]})

    @cherrypy.expose
    def grid(self, **kw):
        """
        Serve up the web page through http://localhost:8000/grid
        """
        return serve_file(os.path.join(os.getcwd(), 'grid.html'))

cherrypy.server.socket_port = 8000
cherrypy.quickstart(Content(),'/')

Note that the CherryPy? and demjson modules are required for this web server to work. He explains the setup in his article.

With the files side by side, you would call the page on port 8000:

http://localhost:8000/grid

So open Windows Task Manager and IE and navigate to this page and watch the amount of memory IE consumes as you scroll through the table.

Also open Firefox and Firebug and watch the grid._by_idx object grow beyond the keepRows setting.

comment:3 Changed 9 years ago by wgrant

I do understand though that is my responsibility for using dojox in my production environment. I just wish I found this sooner.

comment:4 Changed 9 years ago by wgrant

I appears that keepRows is used by the scroller to reuse page objects, but that has little effect on the amount of memory preserved as _Grid still keeps all items in memory.

comment:5 Changed 9 years ago by Adam Peller

Cc: Evan Jared Jurkiewicz ptwobrusell added

dojox projects are rated independently. some are considered stable and are often used in production environments. however, running a scenario like this with a large data set in IE all day long may very well be a problem. I don't have grid-specific advice. You might actually have better luck in the forums with a problem like this. Have you tried using heap tracking tools like sieve? Do they locate the types of objects in memory and what's pinning them down? You might also try in another browser like Chrome using its heap analysis tool and see if the problem occurs across browsers or if it's IE specific. It sounds like you've verified that keepRows is actually set in the grid object. I think the way it's being set on the DOM with dojo.byId rather than a widget instance with dijit.byId is suspect, though even the default of 75 should be a reasonable number.

comment:6 Changed 9 years ago by bill

Owner: changed from bryanforbes to Bryan Forbes

comment:7 Changed 9 years ago by Bryan Forbes

I would be interested to know if this happens with only the data store requesting 1000000 rows of data. The Grid doesn't create copies of the data from the store, only references, so the only memory it uses with regard to data store items is to store the identity and corresponding row number of the data store item. Please re-test just using a data store.

comment:8 Changed 7 years ago by Colin Snover

Priority: highblocker

Bulk update of open ticket priorities.

comment:9 Changed 6 years ago by bill

DojoX Grid and EnhancedGrid are deprecated in favor of dgrid and gridx.

You should upgrade your code to use one of those two grids.

We will consider patches to the old DojoX Grid code though.

comment:10 Changed 6 years ago by bill

Resolution: patchwelcome
Status: newclosed
Note: See TracTickets for help on using tickets.