Opened 9 years ago

Closed 5 years ago

#8578 closed task (wontfix)

Create a means of logging the results of doh test runs to a central location

Reported by: Sam Foster Owned by: Sam Foster
Priority: low Milestone: future
Component: TestFramework Version: 1.5.0b2
Keywords: doh Cc: dante, haysmark
Blocked By: Blocking:

Description

Particularly for rc / beta testing, it would be nice to put a checkout up somewhere and have all test runs log their results back to a server.

The initial plan is to create a patch (for this purpose, not to be committed) for doh/_browserRunner.js, that collects the results, and make a jsonp call with data, including:

  • user-agent
  • tests run/passed/failed/
  • date
  • dojo version

Attachments (4)

8578_browserRunner.patch (3.0 KB) - added by Sam Foster 9 years ago.
configModule.patch (4.7 KB) - added by Sam Foster 8 years ago.
[CLA] [PATCH] Add config module mechanism to doh browser runner, w. remote test result logging
configModule.2.patch (7.0 KB) - added by haysmark 7 years ago.
Added POST support. Expanded results to spell out which assertions failed.
configModule.3.patch (1.4 KB) - added by Sam Foster 7 years ago.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 9 years ago by Sam Foster

Status: newassigned
Type: defecttask

comment:2 Changed 9 years ago by Sam Foster

The initial patch just includes: (e.g.)

  • version 1.3.0dev (16407)
  • startdate 1233792638176
  • ua Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.5) * Gecko/2008120121 Firefox/3.0.5
  • totalTests 769
  • totalGroups 71
  • totalErrors 2
  • totalFailures 1

Need to either collect up all the test group / test names and their status, or get at least the entry point so we can know what tests were being run.

I better way of bolting on this functionality without leaving footprints all over toolkit code would be nice. And similar functionality for rhino/*_runner.js

The xhrGet will likely be replaced by a jsonp call, once the endpoint is setup.

comment:3 Changed 9 years ago by Sam Foster

Modified the patch so it looks for a reportUrl param in the querystring. I'm not sure this is the best way to go about this, but this approach would allow us to commit the patch. OTOH it means you have to ensure the param is there in each link/bookmark/whatever that people use to load up the runner. Hmmm...

Changed 9 years ago by Sam Foster

Attachment: 8578_browserRunner.patch added

comment:4 Changed 9 years ago by Sam Foster

Version: 1.2.31.3.0b1

See http://sfoster.dojotoolkit.org/release/1.3.0b1/ This 1.3.0b1 checkout,patched with the attached _browserRunner patch.

Until we get a report page set up, you can see the data being logged here: http://www.persvr.org/explorer.html - select the DojoTest? store.

The landing page there just lists out all the runTests.html I could find. Any other entry points people would like to see?

comment:5 Changed 9 years ago by Sam Foster

Resolution: fixed
Status: assignedclosed

The patch here was used for the 1.3 release, applies cleanly and provides a reasonable (albeit partly manual) way of enabling this feature on a checkout. Still todo are reporting on the logged data, and potentially applying this patch in a way that allows it to be checked in. These should be seperate tickets

comment:6 Changed 8 years ago by Adam Peller

Milestone: future1.4

comment:7 Changed 8 years ago by Sam Foster

Milestone: 1.41.5
Resolution: fixed
Status: closedreopened
Version: 1.3.0b11.5.0b2

Re-opened as the previous patch no longer applies to trunk. The dohConfig.patch should apply to trunk/util/doh and provides an opt-in way of loading config (aka monkey-patch) modules, including the 'phoneHome' remote logging. In a nutshell:

util/doh/runner.html?configModule=doh.config.phoneHome

How we log test run results needs more work, but this mechanism lets us iterate on that at will.

Changed 8 years ago by Sam Foster

Attachment: configModule.patch added

[CLA] [PATCH] Add config module mechanism to doh browser runner, w. remote test result logging

comment:8 Changed 7 years ago by haysmark

Cc: haysmark added
Component: GeneralTestFramework

I tried experimenting with your patch and the dojox.io.xhrPlugins.fullHttpAdapter doesn't seem very scalable. I tried enhancing the code to return which tests were failing and I got a "413 FULL" message back; it seems fullHttpAdapter was doing a GET after all. Maybe a better approach would be to use either the dojo.io.iframe or even a plain form submit?

comment:9 Changed 7 years ago by Adam Peller

Milestone: 1.51.6

Changed 7 years ago by haysmark

Attachment: configModule.2.patch added

Added POST support. Expanded results to spell out which assertions failed.

comment:10 Changed 7 years ago by haysmark

sfoster, I modified your patch and attached a proof of concept of POSTing the failed assertions. I replace doh._AssertFailure to log the failure message, then create a form to submit the extended results.

comment:11 Changed 7 years ago by Sam Foster

haysmark, I tried your patch and it looks good and works well in my limited testing.

In my latest patch (configModule.3.patch) I just added the ability to have a ';' delimited list of configModules, so you can both phoneHome *and* get Mr. Simpson's commentary on the matter.

Here's a sample result: http://www.persvr.org/DojoTest/[?id=370]

We'll need to purge that db, or find another home when this lands. Probably still need some sample reports to start to get a better idea of what data we should be collecting, and how.

Changed 7 years ago by Sam Foster

Attachment: configModule.3.patch added

comment:12 Changed 7 years ago by bill

Milestone: 1.6future

(sadly) punting seemingly abandoned ticket and meta tickets to future

comment:13 Changed 6 years ago by ben hockey

Priority: highlow

comment:14 Changed 5 years ago by bill

Resolution: wontfix
Status: reopenedclosed

Presumably this will never be done.

Note: See TracTickets for help on using tickets.