Page MenuHomePhabricator

Scap broken for deploying new versions of MediaWiki due to ExtensionMessage file not being created
Closed, ResolvedPublic

Description

Looks like this is related to the improvements Ori and Tim made after the last issues.

When deploying a new version, wmf-config/ExtensionMessages-1.XXwmfY.php isn't created so it can be used

reedy@tin:/a/common$ scap test2wiki to 1.22wmf10 and rebuild l10n cache
Checking syntax... done
Copying to tin from tin.eqiad.wmnet...ok
Done
Updating ExtensionMessages-1.22wmf9.php...
done
Updating LocalisationCache for 1.22wmf9... done
Updating ExtensionMessages-1.22wmf8.php...
done
Updating LocalisationCache for 1.22wmf8... done
Updating ExtensionMessages-1.22wmf10.php...PHP Warning: require(/a/common/php-1.22wmf10/../wmf-config/ExtensionMessages-1.22wmf10.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2632

Warning: require(/a/common/php-1.22wmf10/../wmf-config/ExtensionMessages-1.22wmf10.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2632
PHP Fatal error: require(): Failed opening required '/a/common/php-1.22wmf10/../wmf-config/ExtensionMessages-1.22wmf10.php' (include_path='/a/common/php-1.22wmf10/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.22wmf10:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2632

Fatal error: require(): Failed opening required '/a/common/php-1.22wmf10/../wmf-config/ExtensionMessages-1.22wmf10.php' (include_path='/a/common/php-1.22wmf10/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.22wmf10:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2632
Update of MediaWiki localisation messages failed
sync failed
reedy@tin:/a/common$

Worked around in the laziest way possible - copied the wmf9 file to wmf10

reedy@tin:/a/common$ scap test2wiki to 1.22wmf10 and rebuild l10n cache
Checking syntax... done
Copying to tin from tin.eqiad.wmnet...ok
Done
Updating ExtensionMessages-1.22wmf9.php...
done
Updating LocalisationCache for 1.22wmf9... done
Updating ExtensionMessages-1.22wmf8.php...
done
Updating LocalisationCache for 1.22wmf8... done
Updating ExtensionMessages-1.22wmf10.php...
done
Updating LocalisationCache for 1.22wmf10... done


Version: wmf-deployment
Severity: major
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=60967
https://bugzilla.wikimedia.org/show_bug.cgi?id=63659

Details

Reference
bz51174

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:44 AM
bzimport added a project: Deployments.
bzimport set Reference to bz51174.
Reedy created this task.Jul 11 2013, 3:00 PM
Reedy added a comment.Jul 18 2013, 5:07 PM

Seems ok this week.. :/

ori added a comment.Jul 18 2013, 5:09 PM

(In reply to comment #1)

Seems ok this week.. :/

What was different? Were any new extensions deployed?

Reedy added a comment.Jul 18 2013, 5:12 PM

(In reply to comment #2)

(In reply to comment #1)

Seems ok this week.. :/

What was different? Were any new extensions deployed?

Nothing I don't think. It should create a ExtensionMessages-1.22wmfXX.php each time

Would be great to update this after today's deployment, either as WORKSFORME or as still valid.

Reedy added a comment.Jul 22 2013, 6:47 PM

(In reply to comment #4)

Would be great to update this after today's deployment, either as WORKSFORME
or
as still valid.

We're not deploying a new version, scap won't be used today

So what's the state of this, a few weeks later?

Reedy added a comment.Nov 14 2013, 7:32 PM

This seems to have come back again:

reedy@tin:/a/common$ scap testwiki to 1.23wmf4 and build l10n cache
Checking syntax... done
Copying to tin from tin.eqiad.wmnet...ok
Done
Updating ExtensionMessages-1.23wmf3.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf3... done
Updating ExtensionMessages-1.23wmf2.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf2... done
Updating ExtensionMessages-1.23wmf4.php...PHP Warning: require(/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636

Warning: require(/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636
PHP Fatal error: require(): Failed opening required '/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php' (include_path='/a/common/php-1.23wmf4/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf4:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636

Fatal error: require(): Failed opening required '/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php' (include_path='/a/common/php-1.23wmf4/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf4:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636
Update of MediaWiki localisation messages failed
sync failed
reedy@tin:/a/common$ ./multiversion/activeMWVersions
1.23wmf3 1.23wmf2 1.23wmf4
reedy@tin:/a/common$

greg added a comment.Nov 18 2013, 8:56 PM

Reopened, still highest/major?

Reedy added a comment.Nov 18 2013, 9:21 PM

Not exactly, it's easily "worked around" (make a copy of the previous versions file) I just don't get why it works fine for a few weeks, and then it doesn't. Which is annoying

Reedy added a comment.Feb 6 2014, 4:07 PM

Not seen this for a few weeks... Seems to have come back again today

reedy@tin:/a/common$ scap testwiki to 1.23wmf13 and build l10n cache
[2014-02-06 16:04:31,041] started scap: testwiki to 1.23wmf13 and build l10n cache
[2014-02-06 16:04:31,056] Checking syntax
Copying to tin from tin.eqiad.wmnet...ok
Done
Updating ExtensionMessages-1.23wmf12.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf12... done
Updating ExtensionMessages-1.23wmf11.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf11... done
Updating ExtensionMessages-1.23wmf13.php...PHP Warning: require(/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660

Warning: require(/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660
PHP Fatal error: require(): Failed opening required '/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php' (include_path='/a/common/php-1.23wmf13/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf13:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660

Fatal error: require(): Failed opening required '/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php' (include_path='/a/common/php-1.23wmf13/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf13:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660
Update of MediaWiki localisation messages failed
Traceback (most recent call last):

File "/usr/local/bin/scap", line 220, in <module>
  scap()
File "/usr/local/bin/scap", line 177, in scap
  subprocess.check_call('/usr/local/bin/mw-update-l10n')
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
  raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1

Reedy added a comment.Feb 13 2014, 6:16 PM

Updating ExtensionMessages-1.23wmf14.php...PHP Warning: require(/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673

Warning: require(/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673
PHP Fatal error: require(): Failed opening required '/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php' (include_path='/a/common/php-1.23wmf14/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf14:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673

Fatal error: require(): Failed opening required '/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php' (include_path='/a/common/php-1.23wmf14/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf14:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673
Update of MediaWiki localisation messages failed
18:09:25 ERROR - scap failed: CalledProcessError Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1 (duration: 03m 04s)

Change 113260 had a related patch set uploaded by BryanDavis:
Touch ExtensionMessages-VERSION if not present

https://gerrit.wikimedia.org/r/113260

Change 113260 merged by Ori.livneh:
Add bootstrap initialization for l10n update in new branches

https://gerrit.wikimedia.org/r/113260

Tested fix on tin by renaming ExtensionMessages-1.23wmf14.php to ExtensionMessages-1.23wmf14.php-bak and running mw-update-l10n. ExtensionMessages-1.23wmf14.php was recreated with no diff from backup (and no crash).

The patch seemed to work for the wmf15 rollout, but due to other issues we should probably wait to see what happens with wmf16 before we mark this as resolved.

bd808 added a comment.Feb 27 2014, 8:24 PM

During wmf16 deploy we noticed missing en translations on www.mediawiki.org.

Running a full scap for just wmf16 branch (scap --versions php-1.23wmf16 'rebuild php-1.23wmf16 l10n cache') failed with:

Warning: opendir(/upstream): failed to open dir: No such file or directory in /srv/scap/bin/mergeCdbFileUpdates on line 76
Could not open directory '/upstream'.

This looks to be a bug in the scap-rebuild-cdbs call when versions are specified manually.

Rerunning as a full scap (scap 'full scap; rebuild php-1.23wmf16 l10n cache') performed the scap-rebuild-cdbs step correctly. It also seemed to fix the l10n cache on mw.o.

Change 117154 had a related patch set uploaded by BryanDavis:
Delete stub l10n file after generating ExtensionMessages

https://gerrit.wikimedia.org/r/117154

Change 117154 merged by jenkins-bot:
Delete stub l10n file after generating ExtensionMessages

https://gerrit.wikimedia.org/r/117154

Change 117236 had a related patch set uploaded by BryanDavis:
Remove l10n file as user l10nupdate

https://gerrit.wikimedia.org/r/117236

Change 117236 merged by jenkins-bot:
Remove l10n file as user l10nupdate

https://gerrit.wikimedia.org/r/117236

bd808 added a comment.Mar 6 2014, 8:27 PM

It worked!

bd808 added a comment.Mar 6 2014, 9:11 PM

Or not:

14:03] < bd808> greg-g: Nemo_bis found an l10n bug in wmf17 -- https://www.mediawiki.org/wiki/Special:Preferences#mw-prefsection-betafeatures
[14:03] < huh> [MatmaRex]
[14:04] < MatmaRex> [21:58] <huh> Did I add i18n for sp-contributions-newonly correctly? :/
[14:04] < MatmaRex> [21:59] <huh> It looks ok here http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:Contributions/108.32.45.153
[14:04] < MatmaRex> [21:59] <huh> but broken here https://www.mediawiki.org/wiki/Special:Contributions/PiRSquared17

bd808 added a comment.Mar 7 2014, 5:18 PM

It just dawned on me that rebuildLocalisationCache.php has a --force option. This might be the answer rather than the attempts that I've made to clean up after the stub l10n generation.

Change 117821 had a related patch set uploaded by BryanDavis:
Force l10n rebuild after bootstrapping

https://gerrit.wikimedia.org/r/117821

Change 117821 merged by jenkins-bot:
Force l10n rebuild after bootstrapping

https://gerrit.wikimedia.org/r/117821

bd808 added a comment.Mar 13 2014, 9:24 PM

I'm going to declare victory on this now. I added the --force flag and the only l10n issues that have been reported against the 1.23wmf18 branch were due to an unrelated partial scap failure.

(In reply to Bryan Davis from comment #26)

I'm going to declare victory on this now.

Or not: https://wikitech.wikimedia.org/wiki/Incident_documentation/20140403-Deploy#Bryan

greg moved this task from To Triage to Done on the Deployments board.Jan 8 2015, 5:40 PM