Here is the current progress of the investigation:

• Created a single EspressoTest flow with multiple conditions.
  1. Skip or not to skip the tutorial page
  2. Press back or not to press back to return to the Feed page
  3. Clear or not to clear the search results
  4. Add a proper paused time to fetching data

• Add a custom snap method to capture the current view
  1. For the reference screenshots
  2. For the image comparison

• Add a simple comparison method to do the images comparison
  1. Import the current screenshot and compare with the reference screenshots, and convert it to Bitmap to do the comparison on:
     1. Pixel counts
     2. RGB counts
  2. OpenCV might be another way to do the comparison

For the possible works for the future testing:

1. Use the EspressoTest test cases to capture the reference screenshots
2. Put the reference screenshots to the asset folder
3. Setup a rules for the fault-tolerance on image comparison. e.g.: different date, updated sentences...
4. Might need to do a reset programmatically after the testing.

Waiting until finish the F-board tasks

Change 394735 had a related patch set uploaded (by Cooltey; owner: Cooltey):
[apps/android/wikipedia@master] [WIP] Sample Codes of EspressoTest Screenshots Comparison

https://gerrit.wikimedia.org/r/394735

Ready for review, and the patch contains only the following flow:

Feed Page -> click on search bar -> change search language to test -> search OOO -> finish

Installation steps

1. Do a first run espresso/SampleOfTests.java on a device or an emulator to generate reference images
2. In you device or emulator, use a file manager app to find a folder (located in internal device folder) called /wikipedia-app-tests/
3. Copy the reference images inside the /wikipedia-app-tests/ and put them under the app assets folder assets/espresso/
4. Run the espresso/SampleOfTests.java

Image Comparison

1. Put ScreenshotTools.snap("[YOUR-PAGE-NAME]", getActivity()); to take a screenshot for the current view
2. Use CompareTools.compare(getActivity(), "[YOUR-PAGE-NAME]") to make a image comparison

Update for T181456

The patch now contains a rotateScreen() method allows the process rotate screen while testing.

Also, simply some methods inside the ViewTools.java, and update the capture zone for ScreenshotTools

Hey @cooltey - is this ticket still blocked, or can we move it to an active column?

Hi @Charlotte

The investigation has already finished, the results from me are doing an EspressoTest with a customized screenshots tool + simple images comparison.

The patch is on Gerrit right now: https://gerrit.wikimedia.org/r/#/c/394735/ with a [WIP] tag, maybe I can move it to Github today.

Might need some feedbacks from @Dbrant and @Sharvaniharan

This is the forked version of the Wikipedia app with EspressoTest + ScreenshotsTest + Images Comparison
https://github.com/cooltey/apps-android-wikipedia

After a discussion with @Dbrant, we think we can close this task and decide how we can build the espresso tests to our app.

Will look for some other apps also using the Espresso Tests for automated UI Testing

Found some apps from Github are using Espresso Tests -

• owncloud - https://github.com/owncloud/android
• Evernote/kgb - https://github.com/evernote/kgb
• Small demo app from Google - https://github.com/google/agera
• Small fitness app - https://github.com/mazurio/bodyweight-fitness-android

Here are some sample code of Espresso Tests from Google
https://github.com/googlesamples/android-testing

Also noticed some apps are using this: http://appium.io/
It is also an open source test automation framework for use with native, hybrid and mobile web apps

A note for setting up the patch and testing environment:
https://www.mediawiki.org/wiki/User:CFeng_(WMF)/EspressoTest_with_Screenshots_Comparison

Change 407510 had a related patch set uploaded (by Sharvaniharan; owner: Sharvaniharan):
[apps/android/wikipedia@master] [WIP]Tweaks to the existing espresso set up

https://gerrit.wikimedia.org/r/407510

Change 394735 merged by jenkins-bot:
[apps/android/wikipedia@master] [SPIKE] Sample codes of EspressoTest + screenshots comparison

https://gerrit.wikimedia.org/r/394735

Change 407510 merged by jenkins-bot:
[apps/android/wikipedia@master] Tweaks to the existing espresso set up

https://gerrit.wikimedia.org/r/407510