Progress is in the prototype/testing/ui-test-screenshots branch. So far it:

• Swipes through each onboarding screen (4 screens in total) and takes a screenshot.
• Does this for an iPhone X 13.5 simulator and iPad Pro 9.7 inch 14.3 simulator
• Also does this for both English and German language settings
• Saves screenshots to user's desktop
• Triggers locally from the fastlane command bundle exec fastlane ios screenshots. This runs tests on the new WikipediaScreenshots scheme, which only runs the new WikipediaScreenshots test plan.

Remaining work/improvements:

• Allow list of simulators to be input into the bash script rather than hardcoded
• Add ability to trigger the fastlane lane from CI (Circle CI, GitHub actions, or the build server). Seems like we should allow both one-off triggering from any branch, and maybe run it on a periodic cadence on main if next step can be accomplished
• Keep track of the last successful screenshots, compare new run to old screenshots, and send failure alert if pixels differ
• Send zip file of screenshots somewhere (Slack? Email?)
• Screenshot the login flow per task description

Update - I was able to get these running from an open PR in CircleCI. It takes screenshots and saves them as CircleCI artifacts. The results are here (you can download screenshots from the Artifacts tab):

https://app.circleci.com/pipelines/github/wikimedia/wikipedia-ios/1002/workflows/edb25d58-82b7-4b7d-9544-1fd90b5c03cc/jobs/975

A couple of notes:

• Because we have the "Only build pull requests" project-wide setting set, I'm not sure if this will kick off on a regular cadence for us from main. We might want to monitor this feature request (and consider the mentioned GitHub actions solution).
• Artifacts are accessible for up to 30 days
• I didn't see a way to export the artifacts elsewhere (Slack, email), but there's a way to pull the latest artifacts via a script (see "Downloading all artifacts for a build on CircleCI" here).

Sync notes:

1. It looks the tests did run automatically last night (see https://app.circleci.com/pipelines/github/wikimedia/wikipedia-ios?branch=ui-test-screenshots), on a branch that did not contain a pull request, so I think the first note in the comment above is a non-issue.
2. No need to worry about Slack integration, it's easy enough to pull artifacts from the CircleCI portal.
3. At this point it would be nice to break it off into two separate test flows: One faster-running smoketest that does not take screenshots, that we possibly trigger upon new PR or a merge into main. Then a another set of tests that take screenshots on various devices and languages, compares against gold standard screenshots (committed to the repo) and errors out if any of them are off.

The Ruby gem for comparing images is chunky_png. This article has example code on how to use it:

• https://jeffkreeftmeijer.com/ruby-compare-images/

@zeljkofilipin Awesome, thanks!

I've recently stumbled upon https://percy.io/ "Your all-in-one visual review platform".