Page MenuHomePhabricator

Fix MultimediaViewer scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily
Closed, ResolvedPublic

Related Objects

Event Timeline

Restricted Application added a project: Multimedia. · View Herald TranscriptMar 27 2015, 11:53 AM
Gilles added a subscriber: Gilles.Mar 27 2015, 3:00 PM

Please don't delete them, a ton of work went into them. I've done many rounds of making them less flaky. They still are, and I don't have time to fix them because this isn't part of our quarterly goals which we're trying to wrap up.

Also, a week is insufficient as a way to measure that a job is unsalvageable. For all we know, they're failing because of a real bug, it's just that nobody has had the bandwidth to look at it for a while. You work on browser tests every day, so a week might make sense to you, but for a team like multimedia it wasn't a weekly focus, it's something that happened when we found time.

zeljkofilipin renamed this task from Delete or fix failed MultimediaViewer browsertests Jenkins job to Fix failed MultimediaViewer browsertests Jenkins jobs.Mar 30 2015, 1:58 PM
zeljkofilipin set Security to None.

Ok, looks like somebody cares about browser tests, so we will not delete any jobs. :) Feel fee to reach out to Release-Engineering-Team team if you need help. Also, I have proposed Workshop: Fix broken browsertests/Selenium Jenkins jobs (T94299) for this year's hackathons.

Gilles added a comment.EditedApr 6 2015, 11:57 AM

I've had a look at the failures and so far it looks like a lot of random timeouts. It could just be beta slowness (this is suggested by the few actual under-the-hood requests timing out), as the test we have running against mediawiki.org is very stable in comparison:

https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/

I'll do the stupid approach, which is to increase the timeout values for assertions timing out, we'll see if there's any improvement.

Given that MediaViewer is now in a stable state and not seeing active development, switching all MediaViewer browser tests to targeting mediawiki.org might be something to consider as well.

Gilles claimed this task.Apr 6 2015, 11:59 AM

Change 202024 had a related patch set uploaded (by Gilles):
Increase values for frequent beta browser tests timeouts

https://gerrit.wikimedia.org/r/202024

Change 202024 merged by jenkins-bot:
Increase values for frequent beta browser tests timeouts

https://gerrit.wikimedia.org/r/202024

Tgr added a comment.Apr 6 2015, 2:01 PM

If the test framework supports that option, repeating failed tests and only reporting in case of two successive failures could help to filter out random errors resulting from beta being unstable.

hashar removed a subscriber: hashar.Apr 8 2015, 2:15 PM
Gilles removed Gilles as the assignee of this task.Aug 5 2015, 8:24 AM

I haven't had spare cycles for this in a long time... Un-licking this cookie since I don't think I should really be responsible for Media Viewer maintenance.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptAug 5 2015, 8:24 AM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:25 PM

Mass-removing the Multimedia tag from MediaViewer tasks, as this is now being worked on by the Reading department, not Editing's Multimedia team.

Jdlrobson raised the priority of this task from Normal to High.Sep 24 2015, 3:54 PM
Jdlrobson added a subscriber: Jdlrobson.

I'd say this is high now my team is supposed to look after this extension and is unfamiliar with exactly what it's supposed to do :)

zeljkofilipin lowered the priority of this task from High to Normal.Sep 25 2015, 2:26 PM

@Gilles do you need the mmw.performance browser test still? It doesn't seem to quite work and I'm wondering if it's safe to delete?

It depends on why it fails. It's a safeguard against media viewer's performance becoming worse than the file page's. If it's started failing a lot, it might not be a sign that the test has a problem, it could be that it's doing its job and telling you that media viewer has experienced a performance regression.

The error log clearly shows that it's a syntax problem, probably a change in the framework or cucumber due to an update:

https://integration.wikimedia.org/ci/view/BrowserTests/view/MultimediaViewer/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/819/console

https://github.com/wikimedia/mediawiki-extensions-MultimediaViewer/blob/master/tests/browser/features/step_definitions/mmv_performance_steps.rb#L27

That line was touched a month ago by @zeljkofilipin, apparently related to a change to the framework:

https://github.com/wikimedia/mediawiki-extensions-MultimediaViewer/commit/2c3ac55a673ec5aba0080bb73789f2ea1110bd93

I'd strongly advise against deleting any test anyway. Especially in this sort of situation where it amounts to tearing down a house because you've found a leaking pipe. Whatever amount of work/difficulty encountered trying to fix the test is negligible compared to the amount of work it took to write it.

@zeljkofilipin would you be willing to fix this since I am not too familiar with the inner workings of the framework?

And destroying man-weeks of work out of laziness to actually address a minor upgrade issue.

Can you please truly assume ownership of that code and maintain it, instead of giving up before even trying? Every time I see something Media Viewer-related you guys are looking for someone else to fix things for you, or you just give up without even trying to understand the code. That's fine for the first couple of months of ownership transfer, but now you own it.

It's even disrespectful of the time I spent on Dec 24 figuring out what was wrong. That stuff looks like a trivial upgrade change, but I'm not going to fix it for you.

I'd get it if you tried and then hit a wall, but this isn't even trying. Please realize that this didn't take 5 minutes to write and that you will be very glad if it works once you start modifying Media Viewer and aren't completely sure that you didn't break something. Which is something to expect when you're stumbling around a codebase you newly own. The tests should be the first thing you care about fixing. You'd be shooting yourselves in the foot in the long run deleting tests now, particularly given the simple reason why they've started breaking.

@zeljkofilipin would you be willing to fix this since I am not too familiar with the inner workings of the framework?

@Jdlrobson: I am willing to help you fix it, but I do not think it is a good idea to just fix it myself. If you learn how to fix the tests, you will know how to it yourself the next time. :) Let me know if you would like to pair on it.

greg added a subscriber: greg.Feb 11 2016, 5:26 PM
zeljkofilipin renamed this task from Fix failed MultimediaViewer browsertests Jenkins jobs to Disable MultimediaViewer scenarios that fail at en.wikipedia.beta.wmflabs.org from running daily.
zeljkofilipin renamed this task from Disable MultimediaViewer scenarios that fail at en.wikipedia.beta.wmflabs.org from running daily to Fix MultimediaViewer scenarios that fail at en.wikipedia.beta.wmflabs.org or disable them from running daily.Feb 25 2016, 2:27 PM
zeljkofilipin renamed this task from Fix MultimediaViewer scenarios that fail at en.wikipedia.beta.wmflabs.org or disable them from running daily to Fix MultimediaViewer scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily.

Change 273234 had a related patch set uploaded (by Zfilipin):
WIP Fix browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce

https://gerrit.wikimedia.org/r/273234

zeljkofilipin closed this task as Resolved.Feb 26 2016, 10:34 AM

Resolving this task. I have created separate tasks with specific problems: T128072, T128079.