Page MenuHomePhabricator

Wikitech blank page and no logs with mediawiki 1.29.0-wmf.7
Closed, ResolvedPublic

Description

I deployed 1.29.0-wmf.7 to group1 wikis (including labswiki) at ~20:06, Icinga alerted almost immediately I rolled back at ~20:14 during this time wikitech was displaying a blank page and also sent no logs to logstash:

https://logstash.wikimedia.org/goto/799c3d7ed1a86b535206cffa83094509

Event Timeline

Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 4 2017, 9:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
thcipriani triaged this task as High priority.Jan 4 2017, 10:04 PM

Well it looks like there are logs on silver itself, and this also explains why no other wikis were affected

[Wed Jan 04 20:06:56.757579 2017] [:error] [pid 6141] [client 52.3.127.144:41017] PHP Warning:  require_once(/srv/mediawiki/php-1.29.0-wmf.7/extensions/SemanticForms/Sema
nticForms.php): failed to open stream: No such file or directory in /srv/mediawiki/wmf-config/wikitech.php on line 7                                                      [Wed Jan 04 20:06:56.757620 2017] [:error] [pid 6141] [client 52.3.127.144:41017] PHP Fatal error:  require_once(): Failed opening required '/srv/mediawiki/php-1.29.0-wmf
.7/extensions/SemanticForms/SemanticForms.php' (include_path='/srv/mediawiki/php-1.29.0-wmf.7:/usr/local/lib/php:/usr/share/php') in /srv/mediawiki/wmf-config/wikitech.php on line 7

Adding @Reedy since I noticed he made a change that touched semantic forms: https://github.com/wikimedia/mediawiki-tools-release/commit/72ca89c3db84bb758d39adb950e6427f05761c46

I think I may have found the issue: https://github.com/wikimedia/mediawiki-extensions-SemanticForms/blob/master/RENAMED.txt is the only thing in php-1.29.0-wmf.7 :((

Reedy added a comment.Jan 5 2017, 2:31 AM

This is basically T153257 again.. :(

Easiest "fix" for now, is to manually checkout the 3.7 tag, and push that as a submodule update.... As master was rotting (and emptied as it's actually moved to github) but the tag would work fine for now, rather than changing to using PageForms master

And for future... make-wmf-branch is apparently going away? In the meantime, do we just create a branch with the right name, push to gerrit, so it'll use that?

This is basically T153257 again.. :(

Well, wish I would've seen that one!

Easiest "fix" for now, is to manually checkout the 3.7 tag, and push that as a submodule update

Yup, this is done. Paused the train yesterday post-outage, so I'll full scap again today and should be able to close this.

make-wmf-branch is apparently going away?

@demon and @mmodell have mostly been making the push here. My feeling is that it's pretty close, but we'll be using make-wmf-branch for a few more branches.

In the meantime, do we just create a branch with the right name, push to gerrit, so it'll use that?

Yes please!

thcipriani closed this task as Resolved.Jan 5 2017, 8:21 PM
thcipriani claimed this task.

group1 has 1.29.0-wmf.7 and wikitech yet lives. Closing.

Reedy added a comment.Jan 10 2017, 6:02 PM

FYI, I have created a 3.7 branch...

reedy@ko-kra:~/SemanticForms$ git checkout 3.7
Note: checking out '3.7'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 7e9e627... Version 3.7
reedy@ko-kra:~/SemanticForms$ git checkout -b 3.7
Switched to a new branch '3.7'
reedy@ko-kra:~/SemanticForms$ git push origin refs/heads/3.7:refs/heads/3.7
Total 0 (delta 0), reused 0 (delta 0)
remote: Processing changes: refs: 1, done    
To ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/SemanticForms
 * [new branch]      3.7 -> 3.7