Create a means of logging the results of doh test runs to a central location

[Sam Foster]

Particularly for rc / beta testing, it would be nice to put a checkout up somewhere and have all test runs log their results back to a server.

The initial plan is to create a patch (for this purpose, not to be committed) for doh/_browserRunner.js, that collects the results, and make a jsonp call with data, including:

• user-agent
• tests run/passed/failed/
• date
• dojo version

[Sam Foster]

The initial patch just includes: (e.g.)

• version 1.3.0dev (16407)
• startdate 1233792638176
• ua Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.5) * Gecko/2008120121 Firefox/3.0.5
• totalTests 769
• totalGroups 71
• totalErrors 2
• totalFailures 1

Need to either collect up all the test group / test names and their status, or get at least the entry point so we can know what tests were being run.

I better way of bolting on this functionality without leaving footprints all over toolkit code would be nice. And similar functionality for rhino/*_runner.js

The xhrGet will likely be replaced by a jsonp call, once the endpoint is setup.

[Sam Foster]

Modified the patch so it looks for a reportUrl param in the querystring. I'm not sure this is the best way to go about this, but this approach would allow us to commit the patch. OTOH it means you have to ensure the param is there in each link/bookmark/whatever that people use to load up the runner. Hmmm...

[Sam Foster]

See ​http://sfoster.dojotoolkit.org/release/1.3.0b1/ This 1.3.0b1 checkout,patched with the attached _browserRunner patch.

Until we get a report page set up, you can see the data being logged here: ​http://www.persvr.org/explorer.html - select the DojoTest? store.

The landing page there just lists out all the runTests.html I could find. Any other entry points people would like to see?

[Sam Foster]

The patch here was used for the 1.3 release, applies cleanly and provides a reasonable (albeit partly manual) way of enabling this feature on a checkout. Still todo are reporting on the logged data, and potentially applying this patch in a way that allows it to be checked in. These should be seperate tickets

[Sam Foster]

Re-opened as the previous patch no longer applies to trunk. The dohConfig.patch should apply to trunk/util/doh and provides an opt-in way of loading config (aka monkey-patch) modules, including the 'phoneHome' remote logging. In a nutshell:

util/doh/runner.html?configModule=doh.config.phoneHome

How we log test run results needs more work, but this mechanism lets us iterate on that at will.

[Sam Foster]

[CLA] [PATCH] Add config module mechanism to doh browser runner, w. remote test result logging

[haysmark]

I tried experimenting with your patch and the dojox.io.xhrPlugins.fullHttpAdapter doesn't seem very scalable. I tried enhancing the code to return which tests were failing and I got a "413 FULL" message back; it seems fullHttpAdapter was doing a GET after all. Maybe a better approach would be to use either the dojo.io.iframe or even a plain form submit?

[haysmark]

Added POST support. Expanded results to spell out which assertions failed.

[haysmark]

sfoster, I modified your patch and attached a proof of concept of POSTing the failed assertions. I replace doh._AssertFailure to log the failure message, then create a form to submit the extended results.

[Sam Foster]

haysmark, I tried your patch and it looks good and works well in my limited testing.

In my latest patch (configModule.3.patch) I just added the ability to have a ';' delimited list of configModules, so you can both phoneHome *and* get Mr. Simpson's commentary on the matter.

Here's a sample result: ​http://www.persvr.org/DojoTest/[?id=370]

We'll need to purge that db, or find another home when this lands. Probably still need some sample reports to start to get a better idea of what data we should be collecting, and how.

[bill]

(sadly) punting seemingly abandoned ticket and meta tickets to future

[bill]

Presumably this will never be done.