Opened 8 years ago

Closed 7 years ago

Last modified 6 years ago

#12740 closed enhancement (fixed)

doh.robot alternatives for mobile

Reported by: haysmark Owned by: haysmark
Priority: high Milestone: 1.9
Component: TestFramework Version: 1.6.0
Keywords: Cc: Colin Snover
Blocked By: Blocking:

Description

We need a way to run our existing mobile robot test set on actual mobile devices.

Some people have suggested substituting in synthetic events for the doh.robot API. There are two issues blocking this idea:

  1. API compatibility. Synthetic events are only received by the element you send them to. Most of the doh.robot API does not let you specify the element you want to send events to.
  2. Misinterpretation. If a test driven by synthetic events passes on a mobile device, that really tells you nothing. If you repeat the test's actions manually, the mobile device's default actions can and do cause completely unexpected things to occur. "Trust, but verify" does not work for Dojo because we are the guys that make sure everything works.

We could also look into using Selenium V2, which for each browser supplies a custom plugin/build that listens for actions and executes them semi-natively by calling the native event handlers directly. HOWEVER, the mobile versions of these browsers are not in the official stores; hence, you have to use the SDKs to install them and thus must be working for a company that is willing to agree to the terms of the license agreements.

Another idea is to use the emulators/rooted mobile devices with VNC servers and have the robot act as a server that the tests in turn interact with over XHR. The primary issue here is mobile device performance; the Android emulators are extremely slow, while the VNC servers on mobile devices also bog down the devices if you do not lower their priority. Also, rooting a device is a frowned upon practice in itself.

There is really no clean solution...

Attachments (3)

12740_util.patch (19.5 KB) - added by haysmark 7 years ago.
Bridge API from DOH->WebDriver? with handy form.
12740_dojox.patch (4.8 KB) - added by haysmark 7 years ago.
Mobile tests are currently hosed; get at least one working for demo.
xdomain.patch (760 bytes) - added by haysmark 7 years ago.
WebDriver? patch to trunk/java repo.

Download all attachments as: .zip

Change History (22)

comment:1 Changed 8 years ago by haysmark

In [25896] Added mobile DOH runner. Refactored common js used by both runner.html and mobileRunner.html. Refs #12740.

comment:2 Changed 8 years ago by haysmark

In [26514]:

Disable zoom on mobileRunner.html. Refs #12740. !strict

comment:3 Changed 8 years ago by ben hockey

In [26804]:

get doh/tests/scopeTest and doh/tests/selfTest working.
refs #13956, #13957, #12672, #12740, #12678 !strict

comment:4 Changed 7 years ago by bill

Cc: Colin Snover added

About Selenium, I assume you mean http://code.google.com/p/selenium/wiki/WebDriverForMobileBrowsers. Also to note: javascript API at http://code.google.com/p/selenium/wiki/WebDriverJs.

CC'ing Colin since he said he was interested in mobile testing.

comment:5 Changed 7 years ago by Colin Snover

Thanks for the cc.

Of what exists currently, Selenium appears to be the most promising direction at the moment but there are four issues that I’ve identified (needing an Apple Developer account for iOS is not such a big deal IMO):

  1. The !IPhoneDriver appears to use synthetic events generated within JavaScript, not events from outside the JavaScript sandbox
  2. WebDriverJs doesn’t implement commands for touch events
  3. The mechanism of Selenium is significantly different from the way DOH works now; it uses a remote test runner to send commands to the target browser and then the remote test runner performs the assertion testing based on the returned data. This isn’t bad per se but it does exchange one set of complications for another
  4. Selenium relies on Java, whereas we are trying to move away from that; we could write our own WebDriver server or something but I am not sure that is a good use of manpower

Buster.JS might be more promising in the long term, though I haven’t looked into it terribly deeply to say for sure. Speaking of long term, we need to meet these goals somehow, across all platforms:

  1. Unit testing (of course)
  2. Automated functional testing (next-gen DOH Robot)
  3. Some sort of continuous integration process (I imagine this exists for Selenium but have not found any yet)
  4. Some sort of code coverage analysis mechanism (JSCoverage already works well with DOH)

Changed 7 years ago by haysmark

Attachment: 12740_util.patch added

Bridge API from DOH->WebDriver? with handy form.

Changed 7 years ago by haysmark

Attachment: 12740_dojox.patch added

Mobile tests are currently hosed; get at least one working for demo.

comment:6 Changed 7 years ago by haysmark

I added a DOH plugin replacing the robot with WebDriver? on Android. See the attached patches. I had to fix a tiny bug in WebDriver? so use this APK:

http://haysmark.dojotoolkit.org/android-server.apk

To start a test, on your desktop load util/doh/plugins/android-webdriver-robot.html and follow the instructions.

Changed 7 years ago by haysmark

Attachment: xdomain.patch added

WebDriver? patch to trunk/java repo.

comment:7 Changed 7 years ago by bill

Cool. So the test code all runs in the browser, like before, but it sends commands to WebDriver? by talking to the browser over XHR (to port 8080)? A loopback connection, right?

comment:8 Changed 7 years ago by haysmark

Yes the test code all runs in the mobile browser. It sends commands to WebDriver? as XHRs to 8080 over the loopback connection (hence the need for proper cross-domain scripting). This approach lends to more stable results than Selenium because after the test starts, no commands get sent over wifi.

comment:9 Changed 7 years ago by bill

Makes sense. Plus it doesn't require changing how the tests check the results of the robot operations (i.e. it's easy to examine the state of the DOM and the widgets on the test page).

comment:10 Changed 7 years ago by haysmark

Now that 1.8 is out, do we want to commit this?

comment:11 Changed 7 years ago by bill

Yes, I think so. From the meeting we said that trunk is open for new checkins, except of course backwards compatibility breaking checkins. Anything that you would want to go into a 1.9 release if we eventually have one.

comment:12 Changed 7 years ago by haysmark

Resolution: fixed
Status: newclosed

In [29523]:

Added a WebDriver?/doh.robot bridge for use with Android. See the included HTML file to get started. NOTE: the mobile tests were broken during their AMD refactor. Fixes #12740. !strict

comment:13 Changed 7 years ago by haysmark

In [29524]:

Fix for ButtonList?.html; hopefully gets us closer to a pattern we should be using for AMD robot tests. Refs #12740, #7681.

comment:14 Changed 7 years ago by bill

Milestone: future1.9
Resolution: fixed
Status: closedreopened

Hi. remoteRobot.js is breaking the doc parser because of it's strange syntax. Can't it be changed to a simple define(...) call?

comment:15 Changed 7 years ago by haysmark

Yes, I will see what I can do. This was the format used in doh files in the original AMD conversion.

comment:16 Changed 7 years ago by haysmark

Resolution: fixed
Status: reopenedclosed

In [29571]:

Update old doh AMD style to one that is easier to parse. Fixes #12740. !strict

comment:17 Changed 6 years ago by bill

In [30237]:

Remove some console.log() statements that are breaking optimization with closure compiler (during a build).

Lines like "top.console.log(group);" are converted to "top.0 && console.log(group);" and then the closure compiler complains about missing semicolon before where the && is. Not sure what "top" is, why it's getting converted to "top.0", or why the optimizer is complaining, but seems like the simplest thing is to just remove those lines.

Refs #12740 !strict.

comment:18 Changed 6 years ago by haysmark

In [31077]:

Set EOL style. Refs #12740.

comment:19 Changed 6 years ago by haysmark

In [31207]:

doh reference magically vanished? Refs #12740.

Note: See TracTickets for help on using tickets.