Also see T307073: Job queue backlog time increases for the `wikibase-InjectRCRecords` and `refreshLinks` jobs

This happened again today. The special pages involving categories eventually updated at around 00:30 UTC on May 29th but so far, no updates today on May 31st and they usually also update on the 1st of every month so the reports should have been issued both today and also tomorrow....so we'll see.

If anyone who has control over this procedure is paying attention to this ticket, the 13:00-15:00 UTC time slot that the reports used to be issued was very convenient to the people on en.wiki who work with the reports so it would be awesome if whenever this problem with Special Pages is fixed, they would return to being generated at that time of the day. Thank you.

Update: Okay, so minutes after I posted this, someone flipped a switch or ran a program and the reports were issued. So maybe they don't happen automatically and they need a human being to request them. So, thanks for the reports today, late is better than not at all, but it would be ideal if we could go back to the regular schedule which until recently, was very reliable.

Maybe calling for these reports this was a duty of a staff person who has moved on to a different job or other responsibilities. If someone could add it to their morning to-do list every 3 days (could one ever hope for daily reports?!) that would superb! Thanks again.

Just a reminder that today is June 1st and the Special Pages, at least the ones we utilize on En.Wiki, typically get generated on the 1st of the month...and then the 4th, 7th, 10th, 13th, 16th, 19th, 22nd, 25th, 28th and sometimes the 31st. If they could get back to being generated on their old, regular schedule, that would be great!

Just thought I'd add a little reminder to this Phab ticket. Many thanks!

In T307314#7974768, @Majavah wrote:

Increasing the update frequency by changing it to process one wiki from each section at the same time doesn't seem to be an option either, since there are special pages related to Commons and Wikidata that would then risk overloading those individual wikis.

If it's not reliably finishing in time, I think we should be aiming to parallelize it by shard. Maybe for the main jobs we can exclude the Commons/Wikidata special pages, and then run those as part of the s4/s8 jobs?

Change 804788 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/puppet@production] mediawiki: Split updateSpecialPages.php job to be per-shard

https://gerrit.wikimedia.org/r/804788

With very aggressive rounding up to nearest minute, the enwiki run takes 42 minutes. commonswiki is 8.8 hours, wikidatawiki is ~50 minutes. I'm tagging DBA for a heads-up and review, since this is mostly going to have an impact on databases.

If this per-shard split goes well, I'd like to increase the frequency from 3 days to 2 days and if that goes well, to daily.

@Legoktm this queries would go to the replicas on the slow section for MW right?

In T307314#7997561, @Marostegui wrote:

@Legoktm this queries would go to the replicas on the slow section for MW right?

Yes. Specifically,

	/**
	 * Get a DB connection to be used for slow recache queries
	 * @stable to override
	 * @return IDatabase
	 */
	protected function getRecacheDB() {
		return $this->getDBLoadBalancer()
			->getConnectionRef( ILoadBalancer::DB_REPLICA, [ $this->getName(), 'QueryPage::recache', 'vslow' ] );
	}

(And the one place I saw it overridden in production-deployed code correctly passed 'vslow' as well.)

That's great then. I have lost track of which scripts were changed to reload the config (T298485), is this one of the ones that got that implemented?

My quick skim of that ticket is that reloading config hasn't been implemented in MW yet, unless it was in a patch not linked on that ticket.

Maybe we can create an specific task for that implementation so we don't hijack this one :)

I have been told, probably repeatedly, that Special Pages are generated automatically every 3 days but the ones I regularly use https://en.wikipedia.org/wiki/Special:UnusedCategories and https://en.wikipedia.org/wiki/Special:WantedCategories, were supposed to update some time on August 22nd and didn't.

I'm not sure where to report this so I came back here. Maybe someone can tell me a reason why this happens and when this problem might be resolved by the system.

The automatic update is a systemd timer (formerly 'cron job') that is running on the mw maintenance server.

mediawiki_job_update_special_pages.service is triggered by mediawiki_job_update_special_pages.timer

Checking the status of the service I see this:

Active: active (running) since Fri 2022-08-19 05:00:00 UTC; 4 days ago

And in the process list I can see multiple processes with "updateSpecialPages" in their names. Those are still running since they started on the 19th.

The config for _when_ it runs is *-1/3 05:00 which translates to:

Normalized form: *-*-01/3 05:00:00
    Next elapse: Thu 2022-08-25 05:00:00 UTC
       From now: 1 day 9h left

The underlying problem is that these scripts when getting updated do extremely expensive queries, like full table scans on tables with billions of rows. It will be slightly better after normalization of these tables but clearly mysql is not built to handle queries like this. The reporting need to come from hadoop that is built and architectured towards such large-scale querying. I already created the ticket for it T309738: Move MediaWiki QueryPages computation to Hadoop and if people get enough resources for building the API from AQS or anything like that, I'll do the mediawiki part. Afterwards we can even increase the frequency of the updates and it would remove a massive burden on the core databases.

Change 804788 merged by Dzahn:

[operations/puppet@production] mediawiki: Split updateSpecialPages.php job to be per-shard

https://gerrit.wikimedia.org/r/804788

@Liz the existing job is super close to being finished.. it's currently at zhwikisource going alphabetically. Also we are about to switch this to multiple separated jobs, one per shard.

deployed first https://gerrit.wikimedia.org/r/804788 and then https://gerrit.wikimedia.org/r/c/operations/puppet/+/804800 on mwmaint2002/1002 by @Legoktm

from puppet point of view they are both identical and new services/timers have been created. the old timer has been removed by puppet:

which DC actually runs them is decided in conftool.

[mwmaint1002:~] $ file /lib/systemd/system/mediawiki_job_update_special_pages_*
/lib/systemd/system/mediawiki_job_update_special_pages_s11.service: ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s11.timer:   ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s1.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s1.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s2.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s2.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s3.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s3.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s4.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s4.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s5.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s5.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s6.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s6.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s7.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s7.timer:    ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s8.service:  ASCII text
/lib/systemd/system/mediawiki_job_update_special_pages_s8.timer:    ASCII text

@Liz and all:

We now have one job per shard ,the command lines are:

[mwmaint1002:~] $ for shard in 1 2 3 4 5 6 7 8 11; do grep Exec /lib/systemd/system/mediawiki_job_update_special_pages_s${shard}.service; done
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s2.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s3.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s4.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s5.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s6.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s8.dblist
ExecStart=/usr/local/bin/mw-cli-wrapper /usr/local/bin/mwscriptwikiset updateSpecialPages.php s11.dblist

They all run at the same time, at the old schedule: *-1/3 05:00 and that's ok because gerrit:804800 made it possible to parallelize them. thanks to @Legoktm

And just to spell it out, enwiki is the only wiki in s1, so it should never miss a run because some other wiki has delayed it. Thank you @Dzahn for pushing this forward :)

old (flock) processeds killed. now we are just waiting for the new jobs to run. the scheduled time for s1 (enwiki) is: Trigger: Thu 2022-08-25 05:00:00 UTC; 1 day 6h left

What's the status of this? Is this fixed?