Page MenuHomePhabricator

Performance regression in wmf15 roll-out
Closed, ResolvedPublic

Description

The roll-out of wmf15 to non-Wikipedias appears to have introduced a significant performance regression, which registered as a 1-second increase in median page load times across the cluster. The regression went away when I reverted https://gerrit.wikimedia.org/r/#/c/188409/.

Changelog: https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf15

Event Timeline

ori created this task.Feb 4 2015, 10:43 AM
ori raised the priority of this task from to Unbreak Now!.
ori updated the task description. (Show Details)
ori added subscribers: ori, greg.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2015, 10:43 AM
faidon added a subscriber: faidon.Feb 4 2015, 10:59 AM
Joe added a subscriber: Joe.Feb 4 2015, 1:42 PM
Cmcmahon set Security to None.Feb 4 2015, 3:13 PM
Cmcmahon added a subscriber: Cmcmahon.
hoo added a subscriber: hoo.Feb 4 2015, 3:27 PM
bd808 added a subscriber: bd808.Feb 4 2015, 3:47 PM
scfc added a subscriber: scfc.Feb 4 2015, 4:28 PM
Florian added a subscriber: Florian.Feb 4 2015, 4:31 PM
Matanya added a subscriber: Matanya.Feb 4 2015, 4:37 PM
Anomie added a subscriber: Anomie.Feb 4 2015, 4:48 PM
Legoktm updated the task description. (Show Details)Feb 4 2015, 5:26 PM
Tgr added a subscriber: Tgr.Feb 4 2015, 6:29 PM

Looks like the rollback messed up ResourceLoader caching somehow, see T88576.

bd808 added a comment.Feb 4 2015, 10:18 PM

Group1 was put back on 1.25wmf15 at 2015-02-04T20:33Z. mediaWikiLoadComplete.desktop timings rose again, but the rise started a ~19:40Z and has basically held steady since.

greg added a subscriber: Reedy.Feb 5 2015, 3:12 AM

If all is well over night, the plan is for @Reedy to push out wmf16 to testwikis (group0) and wmf15 to the wikipedias (group1 and group2).

greg closed this task as Resolved.Feb 5 2015, 10:11 PM
greg claimed this task.
20:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf16
20:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf15

Back on track.

I think we look good enough, given everything that's happened today/yesterday with outages and loss of graphite data:


greg reassigned this task from greg to ori.Feb 5 2015, 10:12 PM
Krenair added a subscriber: Krenair.Feb 6 2015, 5:08 PM
Milimetric moved this task from Scheduled to Done on the Scrum-of-Scrums board.Feb 25 2015, 6:56 PM