Opened 12 years ago
Closed 8 years ago
#8578 closed task (wontfix)
Create a means of logging the results of doh test runs to a central location
Reported by: | Sam Foster | Owned by: | Sam Foster |
---|---|---|---|
Priority: | low | Milestone: | future |
Component: | TestFramework | Version: | 1.5.0b2 |
Keywords: | doh | Cc: | dante, haysmark |
Blocked By: | Blocking: |
Description
Particularly for rc / beta testing, it would be nice to put a checkout up somewhere and have all test runs log their results back to a server.
The initial plan is to create a patch (for this purpose, not to be committed) for doh/_browserRunner.js, that collects the results, and make a jsonp call with data, including:
- user-agent
- tests run/passed/failed/
- date
- dojo version
Attachments (4)
Change History (18)
comment:1 Changed 12 years ago by
Status: | new → assigned |
---|---|
Type: | defect → task |
comment:2 Changed 12 years ago by
comment:3 Changed 12 years ago by
Modified the patch so it looks for a reportUrl param in the querystring. I'm not sure this is the best way to go about this, but this approach would allow us to commit the patch. OTOH it means you have to ensure the param is there in each link/bookmark/whatever that people use to load up the runner. Hmmm...
Changed 12 years ago by
Attachment: | 8578_browserRunner.patch added |
---|
comment:4 Changed 12 years ago by
Version: | 1.2.3 → 1.3.0b1 |
---|
See http://sfoster.dojotoolkit.org/release/1.3.0b1/ This 1.3.0b1 checkout,patched with the attached _browserRunner patch.
Until we get a report page set up, you can see the data being logged here: http://www.persvr.org/explorer.html - select the DojoTest? store.
The landing page there just lists out all the runTests.html I could find. Any other entry points people would like to see?
comment:5 Changed 12 years ago by
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
The patch here was used for the 1.3 release, applies cleanly and provides a reasonable (albeit partly manual) way of enabling this feature on a checkout. Still todo are reporting on the logged data, and potentially applying this patch in a way that allows it to be checked in. These should be seperate tickets
comment:6 Changed 12 years ago by
Milestone: | future → 1.4 |
---|
comment:7 Changed 11 years ago by
Milestone: | 1.4 → 1.5 |
---|---|
Resolution: | fixed |
Status: | closed → reopened |
Version: | 1.3.0b1 → 1.5.0b2 |
Re-opened as the previous patch no longer applies to trunk. The dohConfig.patch should apply to trunk/util/doh and provides an opt-in way of loading config (aka monkey-patch) modules, including the 'phoneHome' remote logging. In a nutshell:
util/doh/runner.html?configModule=doh.config.phoneHome
How we log test run results needs more work, but this mechanism lets us iterate on that at will.
Changed 11 years ago by
Attachment: | configModule.patch added |
---|
[CLA] [PATCH] Add config module mechanism to doh browser runner, w. remote test result logging
comment:8 Changed 11 years ago by
Cc: | haysmark added |
---|---|
Component: | General → TestFramework |
I tried experimenting with your patch and the dojox.io.xhrPlugins.fullHttpAdapter doesn't seem very scalable. I tried enhancing the code to return which tests were failing and I got a "413 FULL" message back; it seems fullHttpAdapter was doing a GET after all. Maybe a better approach would be to use either the dojo.io.iframe or even a plain form submit?
comment:9 Changed 11 years ago by
Milestone: | 1.5 → 1.6 |
---|
Changed 11 years ago by
Attachment: | configModule.2.patch added |
---|
Added POST support. Expanded results to spell out which assertions failed.
comment:10 Changed 11 years ago by
sfoster, I modified your patch and attached a proof of concept of POSTing the failed assertions. I replace doh._AssertFailure to log the failure message, then create a form to submit the extended results.
comment:11 Changed 11 years ago by
haysmark, I tried your patch and it looks good and works well in my limited testing.
In my latest patch (configModule.3.patch) I just added the ability to have a ';' delimited list of configModules, so you can both phoneHome *and* get Mr. Simpson's commentary on the matter.
Here's a sample result: http://www.persvr.org/DojoTest/[?id=370]
We'll need to purge that db, or find another home when this lands. Probably still need some sample reports to start to get a better idea of what data we should be collecting, and how.
Changed 11 years ago by
Attachment: | configModule.3.patch added |
---|
comment:12 Changed 10 years ago by
Milestone: | 1.6 → future |
---|
(sadly) punting seemingly abandoned ticket and meta tickets to future
comment:13 Changed 9 years ago by
Priority: | high → low |
---|
comment:14 Changed 8 years ago by
Resolution: | → wontfix |
---|---|
Status: | reopened → closed |
Presumably this will never be done.
The initial patch just includes: (e.g.)
Need to either collect up all the test group / test names and their status, or get at least the entry point so we can know what tests were being run.
I better way of bolting on this functionality without leaving footprints all over toolkit code would be nice. And similar functionality for rhino/*_runner.js
The xhrGet will likely be replaced by a jsonp call, once the endpoint is setup.