Opened 8 years ago

Closed 4 years ago

#8578 closed task (wontfix)

Create a means of logging the results of doh test runs to a central location

Reported by: sfoster Owned by: sfoster
Priority: low Milestone: future
Component: TestFramework Version: 1.5.0b2
Keywords: doh Cc: dante, haysmark
Blocked by: Blocking:

Description

Particularly for rc / beta testing, it would be nice to put a checkout up somewhere and have all test runs log their results back to a server.

The initial plan is to create a patch (for this purpose, not to be committed) for doh/_browserRunner.js, that collects the results, and make a jsonp call with data, including:

  • user-agent
  • tests run/passed/failed/
  • date
  • dojo version

Attachments (4)

8578_browserRunner.patch (3.0 KB) - added by sfoster 8 years ago.
configModule.patch (4.7 KB) - added by sfoster 7 years ago.
[CLA] [PATCH] Add config module mechanism to doh browser runner, w. remote test result logging
configModule.2.patch (7.0 KB) - added by haysmark 7 years ago.
Added POST support. Expanded results to spell out which assertions failed.
configModule.3.patch (1.4 KB) - added by sfoster 7 years ago.

Download all attachments as: .zip

Change History (18)

comment:1 Changed 8 years ago by sfoster

  • Status changed from new to assigned
  • Type changed from defect to task

comment:2 Changed 8 years ago by sfoster

The initial patch just includes: (e.g.)

  • version 1.3.0dev (16407)
  • startdate 1233792638176
  • ua Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.5) * Gecko/2008120121 Firefox/3.0.5
  • totalTests 769
  • totalGroups 71
  • totalErrors 2
  • totalFailures 1

Need to either collect up all the test group / test names and their status, or get at least the entry point so we can know what tests were being run.

I better way of bolting on this functionality without leaving footprints all over toolkit code would be nice. And similar functionality for rhino/*_runner.js

The xhrGet will likely be replaced by a jsonp call, once the endpoint is setup.

comment:3 Changed 8 years ago by sfoster

Modified the patch so it looks for a reportUrl param in the querystring. I'm not sure this is the best way to go about this, but this approach would allow us to commit the patch. OTOH it means you have to ensure the param is there in each link/bookmark/whatever that people use to load up the runner. Hmmm...

Changed 8 years ago by sfoster

comment:4 Changed 8 years ago by sfoster

  • Version changed from 1.2.3 to 1.3.0b1

See http://sfoster.dojotoolkit.org/release/1.3.0b1/ This 1.3.0b1 checkout,patched with the attached _browserRunner patch.

Until we get a report page set up, you can see the data being logged here: http://www.persvr.org/explorer.html - select the DojoTest store.

The landing page there just lists out all the runTests.html I could find. Any other entry points people would like to see?

comment:5 Changed 8 years ago by sfoster

  • Resolution set to fixed
  • Status changed from assigned to closed

The patch here was used for the 1.3 release, applies cleanly and provides a reasonable (albeit partly manual) way of enabling this feature on a checkout. Still todo are reporting on the logged data, and potentially applying this patch in a way that allows it to be checked in. These should be seperate tickets

comment:6 Changed 8 years ago by peller

  • Milestone changed from future to 1.4

comment:7 Changed 7 years ago by sfoster

  • Milestone changed from 1.4 to 1.5
  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Version changed from 1.3.0b1 to 1.5.0b2

Re-opened as the previous patch no longer applies to trunk. The dohConfig.patch should apply to trunk/util/doh and provides an opt-in way of loading config (aka monkey-patch) modules, including the 'phoneHome' remote logging. In a nutshell:

util/doh/runner.html?configModule=doh.config.phoneHome

How we log test run results needs more work, but this mechanism lets us iterate on that at will.

Changed 7 years ago by sfoster

[CLA] [PATCH] Add config module mechanism to doh browser runner, w. remote test result logging

comment:8 Changed 7 years ago by haysmark

  • Cc haysmark added
  • Component changed from General to TestFramework

I tried experimenting with your patch and the dojox.io.xhrPlugins.fullHttpAdapter doesn't seem very scalable. I tried enhancing the code to return which tests were failing and I got a "413 FULL" message back; it seems fullHttpAdapter was doing a GET after all. Maybe a better approach would be to use either the dojo.io.iframe or even a plain form submit?

comment:9 Changed 7 years ago by peller

  • Milestone changed from 1.5 to 1.6

Changed 7 years ago by haysmark

Added POST support. Expanded results to spell out which assertions failed.

comment:10 Changed 7 years ago by haysmark

sfoster, I modified your patch and attached a proof of concept of POSTing the failed assertions. I replace doh._AssertFailure to log the failure message, then create a form to submit the extended results.

comment:11 Changed 7 years ago by sfoster

haysmark, I tried your patch and it looks good and works well in my limited testing.

In my latest patch (configModule.3.patch) I just added the ability to have a ';' delimited list of configModules, so you can both phoneHome *and* get Mr. Simpson's commentary on the matter.

Here's a sample result: http://www.persvr.org/DojoTest/[?id=370]

We'll need to purge that db, or find another home when this lands. Probably still need some sample reports to start to get a better idea of what data we should be collecting, and how.

Changed 7 years ago by sfoster

comment:12 Changed 6 years ago by bill

  • Milestone changed from 1.6 to future

(sadly) punting seemingly abandoned ticket and meta tickets to future

comment:13 Changed 5 years ago by neonstalwart

  • Priority changed from high to low

comment:14 Changed 4 years ago by bill

  • Resolution set to wontfix
  • Status changed from reopened to closed

Presumably this will never be done.

Note: See TracTickets for help on using tickets.