Opened 5 years ago

Closed 4 years ago

#17967 closed defect (patchwelcome)

DOH robot fails to start consistently

Reported by: Terence Kent Owned by:
Priority: undecided Milestone: 1.11
Component: TestFramework Version: 1.9.3
Keywords: Cc:
Blocked By: Blocking:

Description

DOH robot fails to start consistently with (at least)

  • Java 7u55, IE 8, Windows XP
  • Java 7u55, IE 9, Windows 7

This issue is especially significant for new dojo committers (like myself), who want to use the existing tests to confirm patches prior to submitting a pull request.

Background

I recently wanted to submit a few small patches into the dijit package. I setup a few testing VMs and followed the instructions for running the DOH tests, as well the Java 7 specific instructions from the doh/robot/README.md. The tests would never complete successfully, with the doh robot failing to start at random. I eventually reached out to the IRC channel for advice and a user (snover) immediately told me that I had to use a version of Java 6, which appears to be correct. The new VM I setup with Java 6 has been stable so far.

I haven't done any cross testing to figure out if it's all versions of Java 7, or just a windows + Java 7 thing, or whatever, but it's clear that Java 6 is known to make the DOH robot more reliable. IMO, if it turns out that the DOH robot doesn't start up reliably under Java 7, then this really needs to be well documented.

Attachments (1)

doh_results-2014.05.12.tar.bz2 (163.1 KB) - added by Terence Kent 5 years ago.
doh results across multiple runs per vm configuration

Download all attachments as: .zip

Change History (11)

comment:1 Changed 5 years ago by dylan

The plan is to move everything to Intern as fast as we can, https://www.sitepen.com/blog/2014/02/18/dojo-automated-testing-improvements-updating-to-intern/

That said, you pretty much have to use an old version of Java at the moment which is very annoying.

@tkent, care to make a quick pull request against the docs at https://github.com/dojo/docs/blob/master/util/dohrobot.rst (which is used to generate content like http://dojotoolkit.org/reference-guide/1.9/util/dohrobot.html#util-dohrobot )?

comment:2 Changed 5 years ago by Terence Kent

@dylan, Thanks for the quick response. I'd be happy to update the docs, hopefully it will save somebody else time between now and when everything is moved over to Intern. I'll send a pull request over once I've made some changes.

comment:3 Changed 5 years ago by bill

Component: GeneralTestFramework

The tests start fine for me on both Windows XP and Windows 7 with the latest JVM (7.55). I'm doubtful using an old version of the JVM is the answer because then Chrome will refuse to run the JVMwithout clicking a "run this time" button for every test.

You do need to follow the instructions in https://github.com/dojo/util/blob/master/doh/robot/README about setting up certificates. That info should be rolled into the proper manual at https://github.com/dojo/docs/blob/master/util/dohrobot.rst.

comment:4 Changed 5 years ago by Terence Kent

@Bill

Two things:

  1. I did follow the instructions you referenced (Background, sentence 2)
  2. Perhaps I should have been more clear, the tests start successfully, but at random points in the process (typically after 2-3 minutes), the DOH runner won't initialize correctly for every several tests. I've included a paste bin sample of log output here from a Java 7u55 Windows XP/FF system VM, with the certificate accepted. It shows the robot working for some period of time, then just failing to initialize or run. Re-running the again would produce difference failure points.

If you are getting reliable results out of the DOH robot, please do share more information about your hypervisor/VM setup. There is likely something in your setup that makes things more reliable.

comment:5 Changed 5 years ago by bill

I see. I'm getting pretty reliable results for dijit, especially on IE8.

I'm using VirtualBox? because I'm on a mac. Theoretically hypervisor should work too though. Not sure if I have any useful advice, but my rituals include:

  1. (As someone mentioned on IRC) disable mouse integration, i.e. make the VM's mouse independent of your real machine's mouse. But IIRC you said you already did that.
  1. For testing on IE (at least), make sure that the F12 developer tools were never open. Even if they were open and then you closed them, it can still lead to problems (or at least it seemed that way), so I always restart IE in that case.
  1. You might avoid problems by making your VM desktop a standard size, like 1024x768.

Not sure about FF29 since that's fairly new.

comment:6 Changed 5 years ago by Terence Kent

Thanks for posting your config and process. I do need to clarify though - are the tests "pretty reliable" for you, or "completely reliable". I ask because that's the difference between being able to fire off the tests on a few VM's and trust the results of an overall pass/fail, versus only being able to only trust the results of a pass. For any fail scenario, you would then have to go into each failure and see if it's really a failure, or a DOH robot hiccup (repeating the process for each code change).

Assuming your setup is producing "completely reliable" results with Java 7, I decided to do another round of testing with the VMs to be thorough. What I found was, unfortunately, Java 6 vs Java 7 didn't produce drastically different results, both were unreliable.

I've attached the logs from my results so you have some more visibility into what I'm seeing, and here's the an overview of the tests I ran.

  • Parallels 7 used as the hypervisor, Smart Mouse disabled (this is equivalent to VirtualBox?'s "mouse integration")
  • VM Resolution set to 1024x768
  • dojo/dijit/dojox/util master branches used (all up-to-date as of this morning)
  • The dijit/form/runTests.html test suite is ran 3 times
  • Tested with the following combinations
    • Windows 7/IE 9/Java 7
    • Windows 7/IE 9/Java 6
    • Windows XP/IE 8/Java 7
    • Windows XP/IE 8/Java 6

In all cases random robot tests fail. Typically, but not always, failures start on one of the following robot/Spinner_a11y, robot/FormState, or robot/Button_a11y. The robot/validationMessages:after refocus always fails. When ran individually, the tests pass.

Changed 5 years ago by Terence Kent

doh results across multiple runs per vm configuration

comment:7 Changed 5 years ago by bill

Unfortunately the tests have intermittent failures, so if they pass when run individually then you shouldn't worry. Spinner_a11y and validationMessages often fail for me, I guess due to timing conditions. I figure the spinner test is doing something fragile like holding down the button for 1 second and then testing exactly how many times the number was incremented.

comment:8 Changed 5 years ago by Terence Kent

I wish I knew that last week :-). I've spent way too much time trying to figure out what I've been doing wrong to cause the regression tests to fail. I'll update the docs to include a warning that the tests may be unreliable when ran in batch mode, but should be stable ran individually. Hopefully that means that any other test-concious contributor won't follow the same path I did.

I do have to ask then, how do you run a complete set of tests prior to a release? Is there a different method you use in that case and, if so, any chance you could post some details about it? In either case, count me as one of the users waiting on baited breath for the Intern migration.

comment:9 Changed 5 years ago by bill

If some tests fail in batch mode we just run them individually. Intern tests also have spurious failures. It's more a question of how the tests are written and what they are trying to test then what the test harness is.

comment:10 Changed 4 years ago by dylan

Milestone: tbd1.11
Resolution: patchwelcome
Status: newclosed

We have moved dojo core to Intern, and the plan is to focus on Intern rather than maintaining DOH. We would accept pull requests for this issue if someone wants to fix it.

Note: See TracTickets for help on using tickets.