In my previous post fond here: https://quality.mozilla.org/2013/06/robocop-automation-progress/ I talked about the progress made on the Robocop automation. In this post I decided to give an overview of the limitations of the framework. Since we started writing tests we ran into a few issues which we managed to solve. I will be also trying to explain how we managed to solve/bypass this issues.
Limitations of Robocop and where it can be used:
While writing tests we encountered a series of problems so we found a few of the Robocop limitations:
- All Robocop tests will be run on Tegra and Panda boards. For the tests to run correctly we have to make sure it works on phones or tablets running Android 2.3 – this would ensure it runs on the Tegraboards (Android 2.3 and Gingerbread phones UI) – and on 7″ Tablets running Android 4.0 – this would ensure it runs on the Pandaboards (Android 4.0.4 and ICS 7″ Tablet UI).
- No real URLs should be used because the time it takes to load them is not constant, the pages are not static, and they aren’t always available. We can only use local webpages created for the purpose of the test.
- Because it is based on the Robotium test API, it can be used to test the UI elements of Firefox for Android.
- Because it is a UI test tool, it can be used to test basic functional tests, but is not the best tool for performance or endurance testing.
- The Firefox for Android Reader Mode feature is not completely supported yet.
- Robocop receives limited information about the Gecko layer so any Gecko content (Web content) is seen as a picture. You can tap on the content, get the color of a pixel in the content, and drag the content. However, you don’t get any information about the scroll position, whether elements on the page are displayed or have been loaded, whether everything on the page is where it should be, whether the content is correctly displayed, or about what type of content it is (text, picture, empty space, video, flash).
- It works on some devices but not all. This is also caused by the Android fragmentation, the different Android OS versions and builds.
- There are a lot of issues with event timing. The tests will always have to wait for events to happen before doing other actions.
Some of the issues we ran into and who we solved them
We have different UIs for different device types (phones, 7” Tablets and 10” Tablets) and different Android OS versions
- We created 2 support classes: Device to deal with detecting the device type and OS version and Navigation to deal with the UI differences
We had issues with making the tests pass on Tegra and Panda boards
- We investigated and found out that the Tegra boards run Android 2.3 and display the Mobile Android Gingerbread UI and Panda boards run Android 4.1 and display the 7” Tablet UI
On Android 3.0+ pop-up buttons where not accessible if the keyboard is opened
- We have to check the device type and send a BACK key to close the keyboard first
- Recently, since this was not reliable on all devices, created 2 methods in BaseTest: clickOnButton (“label”) which can click the button even if the keyboard is up and toggleVKB() to reliably toggle the position of the keyboard (hidden/unhidden) because using BACK might have minimized the app in a few situations
Some menu items are hidden in submenus (“More” for Android 2.3 and “Tools” for Android 3.0+)
- We created a support method selectMenuItem(“itemName”) that checks for these submenus and selects the item correctly
Some buttons are image buttons it’s hard to click on them
- We are searching for the element after it’s name and clicking the element
The views for Custom Menus, the Settings Menu or Pop-ups are composed of groups of views and we can’t dirrectly access labels or subview types
- We had to dissect each view, check it’s type and advance further if needed
The button position in pop-ups are switched starting from Android 3.0
- We use clickOnButton(name) instead of clickOnButton(index)
We can’t interact with web content
- Some tests had to implement creative solutions like:
- Creating local pages with big links and click in the middle of the page
- Color pages sections in different colors, scroll the page and check the color has changed
- Geolocation local test page changes color if allow or do not allow is chosen
searchText(“text”) does not scroll up if it doesn’t find text scrolling down
- There were situations were we had to search text in the settings menu but the Settings menu was scrolled to a lower possition. We had to do a drag scroll up before search to ensure we found something
The tabs in the Awesomebar (and other views in general) are not always created instantly so getCurrentListViews would not return the correct content
- We had to find some text to wait for in order to wait for the view to be painted correctly
When doing a BACK action if we aren’t where we should be we could minimize the app and crash the test
- We always check when calling a BACK that there is no possibility of minimizing the app – if we are not where we should be the test would fail before the back action