Opened 13 years ago

Closed 13 years ago

Last modified 12 years ago

#2495 closed defect (fixed)

[patch][ccla]flash performance degrades with storage size

Reported by: Adam Peller Owned by: bradneuberg
Priority: high Milestone:
Component: Storage/Flash Version: 0.4.1
Keywords: performance Cc: jcerruti@…, Adam Peller
Blocked By: Blocking:

Description

Information thanks to Julian Cerruti, IBM (CCLA)

We've noticed that the performance - mainly latency - when using dojo.storage.put() with flash suffers noticeably as a direct factor of the amount of data that has been stored. For example on one machine (FF 1.5.0.9 on a T30) storing 18 bytes with a clean store takes about 12ms. The same operation on the same machine after storing 1.7Mb of data takes about 500ms.

I'm attaching two files that comprise a small striped down test case that allows anybody to experiment this problem. In order to use it you will have to modify the .html file to point to the location of a dojo library installed on wherever you want to run it from.

Once you get the HTML working, you can use the buttons above to recreate a situation like the one I decribed. Be careful with the "Clear storage" button, it clears out everything from your dojo storage for the domain you're running.

The "Update flash storage below" button will examine the dojo flash storage and show the information about the number of items stored and the total size of them.

The "Time flash store" button will store a small and a longer item into flash and display how long that took under the current dojo flash state.

You can use the buttons "Add 100 BIG elements" and "Add 1000 SMALL elements" to populate the dojo flash storage artificially.

My recommendation of a test run to see this would be:

0) Open the HTML file, of course and get it to work with your dojo install 1) "Clear storage" 2) "Update flash data below". Make sure it shows 0 items and 0 total size 4) "Time flash store". Note how fast writing 8k is - around 10/20ms, depends on your setup 5) Push the "Add 100 BIG elements" button a few times 6) "Update flash data below" - it will take a while this time. Make sure you have a reasonable value on the total size, something like a couple of Meg 7) "Time flash store". Not how slow it is even to write the 18 bytes now.

With null knowledge about flash and the bridge to dojo, I couldn't help but observe that everything I'm writing to flash through dojo ends up in a single file under a directory with the name of the SWF file where dojo implements the bridge. I was wondering if this could be the cause of the bottleneck. Maybe it could be possible to create a multiplicity of these files on the client when the storage grows too big and then distribute the store requests amont these files. This maybe will help a bit with this problem and help push the limits of this store an additional order of magnitude. Just an idea.

Attachments (7)

test_storage.html (4.6 KB) - added by Adam Peller 13 years ago.
Julian's test case
storage_test.js (10.7 KB) - added by Adam Peller 13 years ago.
Julian's test case
flash_perf.html (1.0 KB) - added by Adam Peller 13 years ago.
Julian's performance test case
flash_perf.js (11.4 KB) - added by Adam Peller 13 years ago.
Julian's performance test case
jcerruti-storage-031.patch (8.8 KB) - added by Adam Peller 13 years ago.
Julian's patch of Dojo 0.3.1
jcerruti-storage-041.patch (7.0 KB) - added by Adam Peller 13 years ago.
Patch against 0.4.1
jcerruti-storage-04x.patch (8.1 KB) - added by Adam Peller 13 years ago.
Patch lifted to latest Dojo 0.4 branch

Download all attachments as: .zip

Change History (15)

Changed 13 years ago by Adam Peller

Attachment: test_storage.html added

Julian's test case

Changed 13 years ago by Adam Peller

Attachment: storage_test.js added

Julian's test case

Changed 13 years ago by Adam Peller

Attachment: flash_perf.html added

Julian's performance test case

Changed 13 years ago by Adam Peller

Attachment: flash_perf.js added

Julian's performance test case

Changed 13 years ago by Adam Peller

Attachment: jcerruti-storage-031.patch added

Julian's patch of Dojo 0.3.1

comment:1 Changed 13 years ago by Adam Peller

Notes from Julian. Patch against 0.3.1 attached, we'll get a 0.4.1 patch as well.

Ok, here's an idea that seems to work for a fix: delay flush()
requests instead of making them synchronously on every "put"
call.

I'm attaching a new Storage.as plus a corresponding .diff file
against the dojo-0.4.1 version of it that depicts the idea.

Note that this is not at all production ready and instead it
only depicts the idea. In order to get this working for everybody
I think at least the following is needed:

- Surfacing the flush() call so that the caller can force flush()
to happen if he wants to make sure nothing is going to be lost
- Allow the caller to specify the amount of time to delay flush()
calls, including "0": don't delay, go back to the old
behaviour and "-1": never flush, I'll flush manually
- Re-place the mechanism where the result of the flush is surfaced
back to the dojo layer - I removed that for the moment
- Properly support dojo storage namespaces by locating and flushing
the proper sharedObject instead of always the default one.

There are probably many other details that I didn't notice. I'll
feel much easier if somebody who know what he/she is doing looks
into this.

Thanks,
~~ Julian

Delaying the flush() seems to avoid the delay somehow. Presumably, it just gets done in the background instead of on a blocking thread. Perhaps this opens up a small window for data loss between the save and the flush?

Changed 13 years ago by Adam Peller

Attachment: jcerruti-storage-041.patch added

Patch against 0.4.1

comment:2 Changed 13 years ago by Adam Peller

Owner: changed from Brad Neuberg to bradneuberg

comment:3 Changed 13 years ago by Adam Peller

Priority: normalhigh
severity: normalcritical

Brad, could you please get this in the 0.4.4 branch and probably the old trunk (seeing as dojo.storage has not yet been ported) so we can close this out before the patch gets stale? Thanks

comment:4 Changed 13 years ago by Adam Peller

Summary: flash performance degrades with storage size[patch][ccla]flash performance degrades with storage size

comment:5 Changed 13 years ago by Adam Peller

Cc: jcerruti@… added; jcerruti@… removed

comment:6 Changed 13 years ago by Adam Peller

Milestone: 0.90.4.4

Changed 13 years ago by Adam Peller

Attachment: jcerruti-storage-04x.patch added

Patch lifted to latest Dojo 0.4 branch

comment:7 Changed 13 years ago by Adam Peller

Resolution: fixed
Status: newclosed

In [8829]

comment:8 Changed 12 years ago by (none)

Milestone: 0.4.4

Milestone 0.4.4 deleted

Note: See TracTickets for help on using tickets.