Page MenuHomePhabricator

MWMultiVersion.php Fatal error: no version entry for `#` (was: special pages has not been updated since November 2019 in jawiki and several other projects)
Closed, ResolvedPublic

Description

Some of the special pages has not been updated since around November 2019 in jawiki. Is there any reason this cannot be updated?

https://ja.wikipedia.org/wiki/%E7%89%B9%E5%88%A5:%E7%89%B9%E5%88%A5%E3%83%9A%E3%83%BC%E3%82%B8%E4%B8%80%E8%A6%A7

特別:更新されていないページ (Special:AncientPages) - Last update 01:02, 22 Nov 2019.
特別:編集履歴の少ないページ (Special:FewestRevisions) - Last update 01:52, 22 Nov 2019.
特別:有効なページへのリンクがないページ (Special:LonelyPages) - Last update 01:16, 22 Nov 2019.
特別:編集履歴の多いページ (Special:MostRevisions) - Last update 01:55, 22 Nov 2019.
特別:被リンクの多いページ (Special:MostTranscludedPages) - Last update 04:58, 22 Nov 2019.

Event Timeline

altocool created this task.Jan 24 2020, 1:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2020, 1:15 PM
Peachey88 added a project: Operations.EditedJan 25 2020, 3:34 AM
Peachey88 added a subscriber: Peachey88.

I think the cron for this group should have run aprox on the 10th & 24th of the month

Dzahn added a subscriber: Dzahn.Jan 25 2020, 3:42 AM

from /var/log/mediawiki/updateSpecialPages.log on mwmaint1001:

17371 jawiki
17372 -----------------------------------------------------------------
17373 jawiki:  Statistics                     [callback] completed in 22.47s
17374 jawiki:  Ancientpages                   [QueryPage] disabled
17375 jawiki:  BrokenRedirects                [QueryPage] got 152 rows in 10.87s
17376 jawiki:  Deadendpages                   [QueryPage] disabled
17377 jawiki:  DoubleRedirects                [QueryPage] got 23 rows in 5.90s
17378 jawiki:  FileDuplicateSearch            [QueryPage] cheap, skipped
17379 jawiki:  ListDuplicatedFiles            [QueryPage] got 3 rows in 0.62s
17380 jawiki:  LinkSearch                     [QueryPage] cheap, skipped
17381 jawiki:  Listredirects                  [QueryPage] got 5000 rows in 0.24s
17382 jawiki:  Lonelypages                    [QueryPage] got 2868 rows in 1m 32.43s
17383 jawiki:  Longpages                      [QueryPage] cheap, skipped
17384 jawiki:  MediaStatistics                [QueryPage] got 8 rows in 0.21s
17385 jawiki:  MIMEsearch                     [QueryPage] cheap, skipped
17386 jawiki:  Mostcategories                 [QueryPage] got 5000 rows in 1m 14.49s
17387 jawiki:  Mostimages                     [QueryPage] got 5000 rows in 10.78s
17388 jawiki:  Mostinterwikis                 [QueryPage] got 5000 rows in 55.21s
17389 jawiki:  Mostlinkedcategories           [QueryPage] cheap, skipped
17390 jawiki:  Mostlinkedtemplates            [QueryPage] got 5000 rows in 51.58s
17391 jawiki:  Mostlinked                     [QueryPage] disabled
17392 jawiki:  Mostrevisions                  [QueryPage] disabled
17393 jawiki:  Fewestrevisions                [QueryPage] disabled
17394 jawiki:  Shortpages                     [QueryPage] cheap, skipped
17395 jawiki:  Uncategorizedcategories        [QueryPage] got 2 rows in 4.63s
17396 jawiki:  Uncategorizedpages             [QueryPage] got 47 rows in 13.83s
17397 jawiki:  Uncategorizedimages            [QueryPage] got 3 rows in 0.74s
17398 jawiki:  Uncategorizedtemplates         [QueryPage] got 777 rows in 0.58s
17399 jawiki:  Unusedcategories               [QueryPage] got 4943 rows in 15.92s
17400 jawiki:  Unusedimages                   [QueryPage] got 5000 rows in 0.32s
17401 jawiki:  Wantedcategories               [QueryPage] got 5000 rows in 14.35s
17402 jawiki:  Wantedfiles                    [QueryPage] got 5000 rows in 27.31s
17403 jawiki:  Wantedpages                    [QueryPage] disabled
17404 jawiki:  Wantedtemplates                [QueryPage] got 5000 rows in 13.33s
17405 jawiki:  Unwatchedpages                 [QueryPage] got 5000 rows in 0.54s
17406 jawiki:  Unusedtemplates                [QueryPage] got 5000 rows in 0.58s
17407 jawiki:  Withoutinterwiki               [QueryPage] got 5000 rows in 0.42s
17408 jawiki:  UnconnectedPages               [QueryPage] cheap, skipped
17409 jawiki:  GadgetUsage                    [QueryPage] got 40 rows in 2m 10.96s
17410 jawiki:  MostGloballyLinkedFiles        [QueryPage] got 0 rows in 0.00s
17411 jawiki:  GloballyWantedFiles            [QueryPage] got 0 rows in 0.00s
17412 jawiki:  DisambiguationPages            [QueryPage] cheap, skipped
17413 jawiki:  DisambiguationPageLinks        [QueryPage] got 5000 rows in 4.14s
17414 -----------------------------------------------------------------

So some of the things you mention are listed as "disabled" but some are not.

I think the cron for this group should have run aprox on the 10th & 24th of the month

Jan 22 22:14 is when updateSpecialPages.log has been written the last time

@Dzahn I guess the question then becomes - why were they disabled?

jijiki triaged this task as Medium priority.Jan 28 2020, 12:56 PM
Dzahn added a comment.Jan 28 2020, 6:29 PM

Looking at what is disabled in general, i see this is not specific to Japanese Wikipedia but globally these are disabled across languages:

Ancientpages
Deadendpages
Mostlinked
Mostrevisions
Fewestrevisions
Wantedpages

Yeah, looks like this didn't run in the four times expected since (two in December, two in January).

Dzahn added a comment.Jan 28 2020, 6:41 PM

Disabling is not at the puppet / cronjob level. These are "sub-jobs" of the single "update_special_pages" job. The script itself is in mediawiki/core, the configuration for it is in mediawiki-config/wmf-config/InitialiseSettings.php.

13263 # Disable all the query pages that take more than about 15 minutes to update
13264 # wgDisableQueryPageUpdate @{
13265 'wgDisableQueryPageUpdate' => [
13266         'default' => [
13267                 'Ancientpages' => 'periodical',
13268                 'Deadendpages' => 'periodical',
13269                 'Fewestrevisions' => 'periodical',
13270                 'Mostlinked' => 'periodical',
13271                 'Mostrevisions' => 'periodical',
13272                 'Wantedpages' => 'periodical',
13273         ],

So the answer to why they are disabled seems to be "because they take more than 15 minutes to update".

Yeah, does whatever in puppet is parsing config not understand the new value?

Dzahn added a comment.Jan 28 2020, 6:47 PM

@Umherirrender Looking at the change above and this ticket, do you know more about why and when these jobs were disabled?

@Umherirrender Looking at the change above and this ticket, do you know more about why and when these jobs were disabled?

The jobs have been disabled for over a decade. This is about the cron replacement for the jobs not running, since the config update.

Dzahn added a comment.Jan 28 2020, 6:54 PM

Yeah, does whatever in puppet is parsing config not understand the new value?

I don't think anything in puppet parses config. From puppet's point of view there is just this:

20     $ensure = mediawiki::state('primary_dc') ? {
21         $::site => 'present',
22         default => 'absent',

and then all the maintenance crons are enabled or disabled depending on "am i in the primary datacenter".

Additionally "update_special_pages" is a single cron job at puppet level and there is no difference between specific special pages.

In puppet is mediawiki::maintenance::updatequerypages::ancientpages which is running the update with the override option to get the special page updated.

It is defined with monthday => [8, 22],

And called from mediawiki::maintenance::updatequerypages

mediawiki::maintenance::updatequerypages::ancientpages { ['s1@11', 's2@12', 's3@13', 's4@14', 's5@15', 's6@16', 's7@17', 's8@18', 'wikitech@19']: }

itself called

# (T17434) Periodical run of currently disabled special pages
class { 'mediawiki::maintenance::updatequerypages': ensure => $ensure }

jawiki lives on s6, which is part of the config. But the update also not running on dewiki, so it seems more than just this maintenance is not running
Please have a look at the logs of that cron job.

The config for DisableQueryPageUpdate is only to show on the special page, the cron jobs must be set up to get the "half-monthly"

# Disable all the query pages that take more than about 15 minutes to update
# Specifiy a "run mode" when a cron job is set up for the special page and
# also create a message for the "run mode" in WikimediaMessages extension with
# the message key 'querypage-updates-<run mode>'
Dzahn added a comment.Jan 28 2020, 9:52 PM

from [mwmaint1002:/var/log/mediawiki/updateSpecialPages/s6@16-AncientPages.log] $

  1 -------------------------------------
  2 #
  3 -------------------------------------
  4 no version entry for `#`.
  5 Fatal error: no version entry for `#`.
  6  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 423
~              

And this Fatal error shows up in all the other shards as well for the same crons.

from [mwmaint1002:/var/log/mediawiki/updateSpecialPages/s6@16-AncientPages.log] $

  1 -------------------------------------
  2 #
  3 -------------------------------------
  4 no version entry for `#`.
  5 Fatal error: no version entry for `#`.
  6  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 423
~              
And this Fatal error shows up in all the other shards as well for the same crons.

There is nothing I can help:

	/**
	 * Get the version as specified in a php file located
	 * at /srv/mediawiki/wikiversions.php.
	 * Result is of the form "php-X.XX" or "php-trunk".
	 * @return String the version directory for this wiki
	 */
	public function getVersion() {
		$this->loadVersionInfo();
		if ( $this->version === false ) {
			self::error( "no version entry for `{$this->db}`.\n" );
		}
		return $this->version;
	}
Koavf renamed this task from special pages has not been updated since November 2019 in jawiki. to special pages has not been updated since November 2019 in jawiki and several other projects.Feb 13 2020, 9:40 PM
Koavf added a subscriber: Koavf.

I can confirm that this is impacting many projects. E.g. https://en.wikipedia.org/wiki/Special:WantedPages and https://www.wikidata.org/wiki/Special:WantedPages and https://species.wikimedia.org/wiki/Special:WantedPages haven't been updated since 2019-11-25 and the latter two claim that they will be updated every two weeks or so (en.wp has a huge warning that it's rarely updated).

This comment was removed by Koavf.
Ciell added a comment.Mar 2 2020, 9:34 PM

Hi all,

Because Dutch Wikipedia will be hitting a mile stone somewhere this month, I'd like to present some nice details of the project to the press, but our Special: pages are also effected by this.
Will this be solved any time soon maybe?

Koavf raised the priority of this task from Medium to High.Mar 2 2020, 9:44 PM

This is not urgent/break now but it's pretty important for maintenance on several wikis and has evidently been broken for four or five months.

Dzahn renamed this task from special pages has not been updated since November 2019 in jawiki and several other projects to MWMultiVersion.php Fatal error: no version entry for `#` (was: special pages has not been updated since November 2019 in jawiki and several other projects).Mar 3 2020, 2:14 AM
Dzahn added a project: Core Platform Team.
Reedy lowered the priority of this task from High to Medium.Mar 3 2020, 2:37 AM

Putting this back at Medium. The underlying issue is fixed, and the scripts will work on the next runs

Reedy added a comment.Mar 3 2020, 2:38 AM

This was only fixed on the 25th Feb, so any maintenance script that hasn't run since, won't have run successfully

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/574726/

Mentioned in SAL (#wikimedia-operations) [2020-03-03T02:39:26Z] <mutante> manually running updateSpecialPages.php maintenance cron on s8 for AncientPages to confirm it was fixed by gerrit:574726 a few days ago (T243599)

Restricted Application added a project: User-Urbanecm. · View Herald TranscriptMar 3 2020, 2:41 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-03T02:57:10Z] <mutante> manually running "Ancientpages" cron on s3 (T243599)