Opened 13 years ago

Closed 13 years ago

Last modified 8 years ago

#2550 closed task (fixed)

implement a test harness for 0.9 which does not rely on a build system

Reported by: alex Owned by: alex
Priority: high Milestone:
Component: TestFramework Version: 0.4.1
Keywords: Cc:
Blocked By: Blocking:

Description

Dojo 0.9 must include a test suite that anyone can run from the command line or from a web browser without any build system or setup other than a checkout of the repository

Change History (42)

comment:1 Changed 13 years ago by alex

(In [7524]) refs #2550

comment:2 Changed 13 years ago by alex

still needs a browser specialization and a browser harness

comment:3 Changed 13 years ago by alex

Status: newassigned

comment:4 Changed 13 years ago by skinner

It would be great if the new test harness made it easy to test dojo functions that operate asynchronously. Back a few weeks ago we made a tentative decision to change the dojo.data APIs so that they support *only* async access, but the current JUM-based unit tests rely mostly on sync access.

Also, it would be great if the new test harness allowed for grouping a set of tests. Ideally a group of unit tests should be able to share some common helper functions, so we can avoid either (a) having the common helper stuff repeated in 10 different tests DRY, or (b) having the ten different tests glommed together into one mega-test, or (c) having the 10 tests rely on helper functions that are defined in a global namespace. (We accidentally got burned a couple times in the dojo.data unit tests, when two different unit test files both defined the same globally scoped helper function.)

comment:5 Changed 13 years ago by alex

(In [7589]) implements async test harness. Refs #2550

comment:6 Changed 13 years ago by alex

(In [7624]) adding getTestCallback() method as discussed with Brian Skinner today in IRC (already doc'd in the porting guide). Almost have the test URL story figured out. Refs #2550

comment:7 Changed 13 years ago by alex

brian,

so you're looking for something like a test suite concept? I guess I didn't implement it based on a felling of YAGNI since you can register anything (including an object, all of whose methods are tests). Would it be sufficient to give test groups setUp and tearDown methods somehow?

comment:8 Changed 13 years ago by skinner

Hey Alex,

Just as an example of what i'm looking for, you might want to check out our existing unit tests for RemoteStore?: http://trac.dojotoolkit.org/browser/trunk/tests/data/core/test_RemoteStore.js

The test_RemoteStore.js file defines 5 functions that are unit tests (the functions with names that start with "test"):

	test_data_core_empty()
	test_data_core_movies_sync()
	test_data_core_movies_async()
	test_data_core_movies_revert()
	test_data_core_movies_save()

The file also includes 3 helper functions, which are used by the unit test functions:

	_test_data_core_newStore()
	_test_data_core_load_movies()
	_test_data_core_movies()

Each of the helper functions is called by two or more of the unit test functions. The helper functions have names that begin with "_" to prevent the test framework from considering them to be test functions and automatically running them.

Unfortunately, the test framework loads this test file into memory at the same time as all the other dojo.data test files, so the helper functions end up defined in the single global namespace that all the dojo unit tests are running in -- to prevent bugs caused by name collisions, the helper functions have painfully long names. The new test harness solves that problem by making it easy to use namespaces like tests.foo.bar.

So what I'm hoping for is just some way of doing the same thing in the new test harness but in a less hackish way -- some way of having a test file that defines 5 actual unit test functions as well as 3 helper functions that are available for the unit test functions to use.

I guess maybe this problem could be solved if a test group had setUp and tearDown functions, as you suggest. Then the setUp function could simply define the 3 helper function. But wouldn't the setUp function end up getting called 5 times, once for each of the unit test functions? That wouldn't be the end of the world, but it would be cleaner if the 3 helper functions only got defined once.

On a related note, here's a guy who argues that you don't really want setUp and tearDown functions at all, just helper functions:

http://www.agileprogrammer.com/dotnetguy/articles/SetupTeardown.aspx

And, also related, here's an article that talks about the usefulness of having a facility for FixtureSetUp? and FixtureTearDown? functions that only get called once for the entire test group, instead of (or in addition to) simple setUp and tearDown functions that get called once per test function. I don't have a ton of experience with unit testing, but to me the idea of FixtureSetUp? and FixtureTearDown? seems more useful than the idea of setUp and tearDown.

comment:10 Changed 13 years ago by alex

(In [7628]) updating with Brian Skinner's excellent suggestions. Refs #2550

comment:11 Changed 13 years ago by alex

(In [7631]) changing the names to assert* from is*. Refs #2550

comment:12 Changed 13 years ago by alex

(In [7634]) un-eff the browser harness in IE. Refs #2550

comment:13 Changed 13 years ago by alex

(In [7635]) ensuring that logging output shows up in the log tab. Refs #2550

comment:14 Changed 13 years ago by alex

(In [7646]) port the command-line tests to run using the most primordial Dojo package loader. Refs #2500. Refs #2550

comment:15 Changed 13 years ago by alex

(In [7647]) keep the test system from conflicting w/ Dojo when run in a browser environment. Refs #2550

comment:16 Changed 13 years ago by alex

(In [7653]) ensure that all tests get loaded in from tests/_base.js. Refs #2550

comment:17 Changed 13 years ago by alex

(In [7654]) ensure that the console is cleared for each test run. Refs #2550

comment:18 Changed 13 years ago by alex

(In [7655]) ensure that the play/pause button is in the right state. Refs #2550

comment:19 Changed 13 years ago by alex

(In [7697]) start logging error objects directly. Helps us get actual failure info out of exceptions. Huzzah. Refs #2550

comment:20 Changed 13 years ago by alex

(In [7702]) fix test-name-finding on Opera. Refs #2550

comment:21 Changed 13 years ago by alex

(In [7703]) get debugging output working correctly on the webkit nightlies. Refs #2550

comment:22 Changed 13 years ago by alex

(In [7704]) output how many tests need to be run. Refs #2550

comment:23 Changed 13 years ago by alex

(In [7847]) adds support for:

  • registering URLs
  • loading of tests in loaded URLs
  • automatic timeouts for loaded URLs
  • automatic test creation from code strings
  • improvements in use with dojo
  • group-level setUp() and tearDown()

Also includes myriad bug fixes. Refs #2550. Still need to document many of these changes in the porting guide.

comment:24 Changed 13 years ago by alex

(In [7879]) keep the system from throwing errors when a test page that includes dojo but doesn't have a parent harness loads it. We still get success/failure info at the console. Refs #2550

comment:25 Changed 13 years ago by alex

(In [7880]) more fixes to prevent errors with test pages not hosted in harness. Refs #2550

comment:26 Changed 13 years ago by alex

(In [7887]) adding sounds so that success and failure sound like...well...success or failure. Refs #2550

comment:27 Changed 13 years ago by alex

(In [7889]) we probably don't want selfTest to always be on when not loading Dojo. Refs #2550

comment:28 Changed 13 years ago by alex

(In [7890]) ensure that we actually play the "woohoo!" sound if it all goes well. Rate limits sounds. Refs #2550

comment:29 Changed 13 years ago by alex

(In [7892]) creating t, f, and is aliases for assertTrue, assertFalse, and assertEqual (respectively). I'm getting really freaking sick of typing the long names. Refs #2550

comment:30 Changed 13 years ago by alex

(In [7893]) a basic summarizer. Refs #2550

comment:31 Changed 13 years ago by alex

(In [7900]) ensure that we dont' fail when loaded stand-alone. Refs #2550

comment:32 Changed 13 years ago by alex

(In [7901]) prevent us from bombing out should we be loaded stand-alone. Refs #2550

comment:33 Changed 13 years ago by alex

(In [7919]) ensure that object comparison tests can end successfully. Refs #2550

comment:34 Changed 13 years ago by alex

(In [7990]) unfscking log output for Webkit and Opera. Refs #2550

comment:35 Changed 13 years ago by alex

(In [7991]) make the log output slightly smaller. Refs #2550

comment:36 Changed 13 years ago by alex

(In [7999]) ensure that paused groups resume correctly. Refs #2550

comment:37 Changed 13 years ago by alex

Resolution: fixed
Status: assignedclosed

Marking fixed. Further improvements to the system should happen under separate bugs.

comment:38 Changed 13 years ago by alex

(In [8155]) now that we've got lots of test groups, roll them up but make them expandable. Refs #2550

comment:39 Changed 13 years ago by (none)

Milestone: 0.9M1

Milestone 0.9M1 deleted

comment:40 Changed 12 years ago by bill

(In [9775]) Turn off sounds by default. Refs #2550.

comment:41 Changed 8 years ago by bill

In [26202]:

fix logic from [25888], it was resolving the Deferred twice, refs #2550 sort of

comment:42 Changed 8 years ago by bill

In [26268]:

Fix logic from [25888], it was resolving the Deferred twice, refs #2550 sort of. Also added some missing d.getTestErrback() calls.

Note: See TracTickets for help on using tickets.