Opened 10 years ago

Closed 10 years ago

Last modified 7 years ago

#9234 closed defect (fixed)

Performance framework extensions for D.O.H.

Reported by: Jared Jurkiewicz Owned by: alex
Priority: high Milestone: 1.4
Component: TestFramework Version: 1.3.0
Keywords: Cc: Eugene Lazutkin
Blocked By: Blocking:

Description (last modified by Jared Jurkiewicz)

This has come about from questions on how well apis like GFX and GFX3D perform. Rough numbers on drawing lines, objects, arcs, in various vector graphics engines.

More generically, it would be great if we could do performance tests in DOH for general APIs and such.

I started talking to Slightlyoff about this and he liked the idea too.

The current, very rough, thoughts are to add a new register function, like:

doh.registerPerformanceGroup(), that would add in properties and functions to a test group related to performance, like min iterations, etc. And with new test types like this, we could have histograms, charts, etc, showing the performance of APIs.

This ticket is to hash out ideas on this and get something implemented!

I'm also thinking the performance targeted tests will need to be able to do some construction outside of the test (similar to setUp), that lets you construct data objects to hand to the task, like draw surfaces and the like.

Attachments (5)

PerfEnhancementsDOH.zip (13.1 KB) - added by Jared Jurkiewicz 10 years ago.
Updated runner.js. Got rid of the recursion, which was causing stack grief on some browsers.
DOHPerfUpdates_Pass2.zip (164.8 KB) - added by Jared Jurkiewicz 10 years ago.
DOHPerformanceTests_pass3.zip (169.7 KB) - added by Jared Jurkiewicz 10 years ago.
doh_perfFixtures.patch (27.4 KB) - added by Jared Jurkiewicz 10 years ago.
Enhancements to DOH to allow for performance tests.
dojox.gfx.perftests.patch.gz (143.8 KB) - added by Jared Jurkiewicz 10 years ago.
Patch with a set of 29 performance tests for GFX primitive operations

Download all attachments as: .zip

Change History (20)

comment:1 Changed 10 years ago by Jared Jurkiewicz

Description: modified (diff)

comment:2 Changed 10 years ago by Jared Jurkiewicz

Okay, I've modified runner.js so that DOH has some basic understanding of 'performance' tests. The modifications allow users to register an entire test group as a performance test set, The contents of the register are exactly the same as if you were registering regular tests, you just pass an additional "perf" parameter to the doh.register() function. If that is set, it will update all test fixures in the register so that they are recognized as performance tests and executed accordingly. Namely,it sets the following properties:

trialDuration: 100, How long you want keep iterating the test function to get measurement data. Default is 100ms. The perf fixture runner will try to calculate how many iterations of the test function must be run to get to that duration. It will then use that # of iterations for all trials.

trialIterations: 10, How many trials are executed of the number of iturations calculated to reach trialDuration

trialDelay: 30 How long do you want to pause between trial executions. This allows for GC time.

Note that if you use the full object test form, you can configure all these options on the actual test, so you can set how long you want your durations to be, now long the delay between trials, and how many tries to run.

Also note that the test results are stored off the root doh object as:

doh.perfTestResults[group][testName]. Where group is the collection of tests, and testname is the name of the test in the group.

Also note you can register performance tests from URLS same as you can regular tests.

The code is by no means perfect yet (and is quite ugly to handle dealing with async tests), but seems to be working well. I would like others to look at this, make suggestions and hopefully we can solidify this enhancement to doh so that module owners can also do performance testing if thet like.

The zip contains the following:

runner.js: An updated DOH runner. perf.js: A test set of 'performance' functions. query_performance.html: Demonstration of registering tests at urls/in other files.

Just extract all those into your util/doh, then you can execute the perf functions by: file:///<your path to dojo>/util/doh/runner.html?testModule=util.doh.perf

comment:3 Changed 10 years ago by Eugene Lazutkin

Cc: Eugene Lazutkin added

Changed 10 years ago by Jared Jurkiewicz

Attachment: PerfEnhancementsDOH.zip added

Updated runner.js. Got rid of the recursion, which was causing stack grief on some browsers.

comment:4 Changed 10 years ago by Jared Jurkiewicz

Okay, so the latest update clears out the recursive iteration for favor of a state based loop. This fixed stack blowing problems on Safari 3.1 and such. Theoretically also should improve timing #s slightly as it removes a function call (stack push, then return). But not all that much, the stack pust + ret is only a few CPU instructions and I think I measured it once at like 0.00007ms or somesuch.

Changed 10 years ago by Jared Jurkiewicz

Attachment: DOHPerfUpdates_Pass2.zip added

comment:5 Changed 10 years ago by Jared Jurkiewicz

The updated zip improves the calibrate function a bit and provides real performance tests for a lot of gfx.

Changed 10 years ago by Jared Jurkiewicz

comment:6 Changed 10 years ago by Jared Jurkiewicz

Pass 3 integrates in dojox.charting for the results! Whoo! Check it out.

Changed 10 years ago by Jared Jurkiewicz

Attachment: doh_perfFixtures.patch added

Enhancements to DOH to allow for performance tests.

Changed 10 years ago by Jared Jurkiewicz

Patch with a set of 29 performance tests for GFX primitive operations

comment:7 Changed 10 years ago by Jared Jurkiewicz

kay, I've gotten several people to look at it and Pete Higgins likes it. So I'm going to commit in the basic updates + example perf tests (in gfx), before something happens to my local copy. It's still very much open for comments/feedback, but as far as I can determine it does not affect any current UT.

I'm doing some final testing now cross browsers before commit.

comment:8 Changed 10 years ago by Jared Jurkiewicz

Tested on:

IE 7

IE 6

Firefox 3

Firefox 2

Safari 3.2.3

Google Chrome 1.0

Opera 9.62

I ran both the perf tests and core dojo doh tests.

No new failures caused by my DOH updates (PERF are distinct and have their own run function over the basic DOH fixture).

comment:9 Changed 10 years ago by Jared Jurkiewicz

Also tested IE 8

comment:10 Changed 10 years ago by Jared Jurkiewicz

(In [17609]) Adding in basic performance module support. \!strict refs #9243

(In [17610]) Adding in basic performance module support. \!strict refs #9243

comment:11 Changed 10 years ago by Jared Jurkiewicz

Accidentally transposed the tracker number. Copied over the info.

IE8 itself seems to have issues with its VML implementation and performance in general. Nothing related to the perf framework, just IE 8. Whee.

comment:12 Changed 10 years ago by Jared Jurkiewicz

Actually, it didn't have issues, IE is just sloooooooooooooow.

comment:13 Changed 10 years ago by Jared Jurkiewicz

(In [17659]) Left a console.log in, as well as it was importing charting when it didn't need to (nothing to chart). \!strict refs #9234

comment:14 Changed 10 years ago by Jared Jurkiewicz

Milestone: tbd1.4
Resolution: fixed
Status: newclosed

Changing milestone since this is checked in and generally working. Also going to close it.

comment:15 Changed 7 years ago by bill

In [27671]:

Fix error about plotResults() being undefined when you run a test on dojo/domReady! rather than on dojo/ready.

Also fixing a number of missing semicolons and other lint issues.

Refs #9234 !strict.

Note: See TracTickets for help on using tickets.