Page MenuHomePhabricator

[EPIC] Run browser tests against every MobileFrontend/Gather patch
Closed, ResolvedPublic

Description

As a Web developer I want browser tests to be run against every change to MobileFrontend and Gather so that I can feel more confident about the quality of the code that I'm merging.


This'll require the following infrastructure:

  1. A bot that listens whatever Gerrit API there is for new patches, runs a suite of browser tests, and reports the results
  2. A server that we run the browser tests on, which hosts an instance of MediaWiki with all the appropriate extensions installed

We've talked with releng and we know that they are working on a generic solution, but it will take a lot of time. We need a temporary solution so we knowingly accept the cost of maintaining our own browser testing infrastructure but we feel that it's far outweighed by the increased confidence in our code. (Our products are very user exposed and we need a good integration tests story).


Outcomes:

  • Browser tests are run on MobileFrontend and Gather on each patchset and get a report of the failing browser tests.
  • A testing environment is set up with mediawiki and our extensions, and the appropriate scripts for triggering the tests.
  • What happens in the testing environment is easily accessible by engineers (transparency about what is going on) (log everything and expose logs).

Related Objects

StatusSubtypeAssignedTask
Resolvedzeljkofilipin
Resolvedzeljkofilipin
ResolvedJhernandez
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
Duplicatephuedx
ResolvedJdlrobson
ResolvedJdlrobson
DuplicateNone
Resolvedbmansurov
ResolvedJdlrobson
ResolvedJdlrobson
ResolvedJdlrobson
DeclinedNone
InvalidNone

Event Timeline

phuedx created this task.May 25 2015, 12:50 PM
phuedx raised the priority of this task from to Needs Triage.
phuedx updated the task description. (Show Details)
phuedx added a subscriber: phuedx.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 25 2015, 12:50 PM
phuedx updated the task description. (Show Details)May 25 2015, 1:24 PM
phuedx set Security to None.
phuedx updated the task description. (Show Details)May 25 2015, 1:28 PM
Jdlrobson moved this task from Needs triage to Product triage on the Gather board.May 25 2015, 9:19 PM
Jhernandez added a subscriber: Jhernandez.
Jhernandez updated the task description. (Show Details)Jun 2 2015, 9:41 AM
Legoktm added a subscriber: Legoktm.Jun 2 2015, 7:53 PM

This'll require the following team-specific infrastructure:

Why is this team-specific? Surely every team with browser tests would eventually like them to be running on every patch.
Why is the wheel being re-invented? T101068 and T101069 are both already implemented by zuul and jenkins, T101063 is done by the CI infrastructure for qunit, and might need some adjustments for browser tests.

Jdlrobson added a subscriber: Jdlrobson.

@Legoktm hopefully discussion on T101325 will help. From talking to Release engineering the plan is to have this for every project, but we want to pioneer and explore getting this now as this seems to be a year or so away in the roadmap.

Update: So if we get the labs instance and the Gerrit account, I estimate writing the script to take no more than an afternoon (all it should involve is something to watch changes and check them out and then run tests and then use gerrit.py for results - this part is mostly done in T101346)
Seems there is a little resistance on e-mail thread that we need to take into account before doing this.

phuedx updated the task description. (Show Details)Jun 10 2015, 11:36 AM

Why is this team-specific? Surely every team with browser tests would eventually like them to be running on every patch.

I've removed that awkward/incorrect wording from the description. Ta.

Jdlrobson moved this task from Product triage to Must haves on the Gather board.Jun 10 2015, 7:24 PM

@Legoktm hopefully discussion on T101325 will help. From talking to Release engineering the plan is to have this for every project, but we want to pioneer and explore getting this now as this seems to be a year or so away in the roadmap.

Didn't really help. Why is the wheel being re-invented? Lets start with a) why is zuul not sufficient for talking to gerrit?

I'm not sure what is being reinvented here. We run browser tests daily on beta labs on master. We want to run browser tests per commit on a non beta labs instance. As far as I understand there is no way to spin up an instance for a given commit and run browser tests. If that's not true that's great and please point me @ relevant docs.

Maybe we should chat over this in irc/hangout if there are some crossed wires.

I'm not sure what is being reinvented here. We run browser tests daily on beta labs on master. We want to run browser tests per commit on a non beta labs instance.

I commented on T101069#1327976, but was told to go here...the task says "When a patch is submitted to Gerrit trigger a script". This is literally what zuul+jenkins do. When a patch is submitted to a specific repository, zuul in turn triggers a jenkins job, which runs whatever defined shell code the job definition says. So, why is that solution inadequate to the point where the wheel needs to be re-invented?

As far as I understand there is no way to spin up an instance for a given commit and run browser tests. If that's not true that's great and please point me @ relevant docs.

When you say instance, do you want a brand new labs instance for each commit? If so, why? Is just a new mediawiki installation sufficient?

Sorry! I've updated the title - it was maybe confusingly worded - yes zuul+jenkins could be the solution to that - we've not investigated the solution to it yet. A new mediawiki instance should be fine - the browser tests only have a few dependencies needed - such as Echo extension and Thanks.

Sounds like we could use zuul in some way - I'm not too familiar with the how yet though so any help fleshing out how we can achieve the task in the main description would be much appreciated.

Please see [Update] Browser tests per patch -email.

Jhernandez triaged this task as High priority.Jun 30 2015, 12:06 PM
Jhernandez updated the task description. (Show Details)

@Jdlrobson Hey I remember you told me the bot was running now automatically picking up new patches on gerrit.

Is that true? What else are we missing? (I'm creating cards for the next sprint https://phabricator.wikimedia.org/tag/reading-web-sprint-51-x/). The main thing I can think of is expose the bot logs somewhere so that we can check what it's doing without having to log into the instance. What do you think?

I know that @dduvall has been experimenting around this and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

It would be worth checking in with him, but the only thing outstanding in terms of my script if we want/need to continue down that road is:

  • Make the bot real time and run on submitting a patch rather than potentially up to 30 minutes delay
  • Post results somewhere and link to it from Gerrit review
  • Make it easier to spin up an instance with Barry - e.g. vagrant role

I know that @dduvall has been experimenting around this and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

Created T104840 for us to actually devote time to that this sprint.

It would be worth checking in with him, but the only thing outstanding in terms of my script if we want/need to continue down that road is:

  • Make the bot real time and run on submitting a patch rather than potentially up to 30 minutes delay

I'm guessing this is T101069, so - tracked ☑️.

  • Post results somewhere and link to it from Gerrit review

Created T104841 to track this.

  • Make it easier to spin up an instance with Barry - e.g. vagrant role

Created T104842 for tracking automation. All this will also be part of how we resolve T104562 as part of making this easily reproducible.

I know that @dduvall has been experimenting around this and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

Created T104840 for us to actually devote time to that this sprint.

Just an update on the mwext-mw-selenium JJB job that it now supports video recording of the Xvfb session (woot): Video files are saved as Jenkins build artifacts for each failed scenario—removed after 3 days to save disk space.[1][2]

The job seems fairly stable (there was a small fix to the supporting slave scripts today, but the MVP in T103039 is now considered done) and is already scheduled under the experimental pipeline for MobileFrontend. Scheduling it for Gather or any other MW extension that uses MW-Selenium 1.4 would only a matter of one line in the JJB/Zuul config.

What other functionality are you looking to achieve with Barry, and can we maybe pool our efforts to bring it into the JJB builder? AFAICT, the triggering and reporting aspects detailed in T101069 are already covered by the latter given that it's run as a Zuul-scheduled Jenkins job, but I could be missing other aspects of the project.

[1] https://gerrit.wikimedia.org/r/#/c/227584/
[2] https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests#Video_recordings_and_screenshots

Ping @Jdlrobson ^. What do we use barry for that is not done by JJB? At what point would it make sense to use JJB instead of barry?

The video recording sounds crazy awesome hah.

Yup let's explore it - T107587
Provided we can cover all the tests I think we are good to scrap Barry in existing form and use him for something else :).

Jdlrobson moved this task from Backlog to Epics on the MobileFrontend board.Aug 4 2015, 6:37 PM

Been pinging around to get resolution with the last two tasks blocking. It'd be great to get this one out of the way already.

Jdlrobson closed this task as Resolved.Sep 8 2015, 11:30 PM
Jdlrobson claimed this task.

I'm calling this done and declaring victory given we've taken this task as far we can.
Good work team!

phuedx added a comment.Sep 9 2015, 8:07 AM

Venimus vidimus vicimus!